Presenting: Drama Harvester

https://fsdfsd.net/J/DramaHarvester

I was thinking about having several bots, but didnt want to hammer the rdrama API so I split the api querying part of my application off.

Its designed to publish events to redis maintaining an nosqlish database of the retrieved items set to time out in 1 hour.

You could subscribe pretty much as many apps as you want to it and that way only one thing is hitting rdrama at a time

Its also leveraging bottleneck, so if you use bottleneck in your API calls back to rdrama and configure it to use the redis instance, you wont exceed rate limiting, and all your applications will share bandwith nicely

I might grab posts or other things, for now Its just focused on comments.

Efficient Data Retrieval: Fetches comments and posts directly from rDrama's APIs, optimized for minimal bandwidth usage.
Redis Integration: Utilizes Redis for caching content, significantly enhancing data retrieval speed and reducing API call redundancy.
Incremental Fetching: Employs smart fetching strategies to avoid duplicate data retrieval, ensuring that only new or updated content is processed.
Scheduled Fetching: Automatically executes data retrieval operations at configured intervals, enabling up-to-date synchronization with rDrama content.
Graceful Shutdown: Implements robust error handling and graceful shutdown processes, ensuring data integrity and application stability.
Scalable Architecture: Designed with scalability in mind, allowing for easy horizontal scaling to accommodate growing data volumes.