Mercurial > hg > Feedworm

annotate feedupdate-main.py @ 81:050748b596a8

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
add another idea that has been going round in my head for some time
author Dirk Olmes <dirk@xanthippe.ping.de>
date Thu, 19 Aug 2010 16:06:18 +0200
parents d20e99d46d78
children e50d446f9942 bf22b32ae9f7
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
2
8a624ee48a74 First skeleton for sqlalchemy: define the mapping and create the first feed
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
1 #!/usr/bin/env python
8a624ee48a74 First skeleton for sqlalchemy: define the mapping and create the first feed
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
2
34
5813e3c10f14 move the database logic out into its own module. Make everything reload safe so that multiple sessions can be created from interactive sessions
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 14
diff changeset
3 import Database
68
525a52169f60 Make FeedEntry table object available in Mapping to use it to expire old, read feed entries.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 64
diff changeset
4 from datetime import datetime, timedelta
2
8a624ee48a74 First skeleton for sqlalchemy: define the mapping and create the first feed
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
5 from Feed import Feed
35
aaec263f07ca Feeds manage the point in time when the next update should happen. FeedUpdater only updates feeds that are due.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 34
diff changeset
6 import FeedUpdater
11
e87c54b3a216 use the logging framework for printing messages
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 6
diff changeset
7 import logging
68
525a52169f60 Make FeedEntry table object available in Mapping to use it to expire old, read feed entries.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 64
diff changeset
8 import Mapping
525a52169f60 Make FeedEntry table object available in Mapping to use it to expire old, read feed entries.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 64
diff changeset
9 from sqlalchemy.sql import and_
34
5813e3c10f14 move the database logic out into its own module. Make everything reload safe so that multiple sessions can be created from interactive sessions
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 14
diff changeset
10 import util
2
8a624ee48a74 First skeleton for sqlalchemy: define the mapping and create the first feed
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
11
11
e87c54b3a216 use the logging framework for printing messages
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 6
diff changeset
12 logger = logging.getLogger("feedupdater")
e87c54b3a216 use the logging framework for printing messages
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 6
diff changeset
13
4
e0199f383442 retrieve a feed for the given URL, store entries as feed_entry rows into the database
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 2
diff changeset
14 def listFeeds(session):
e0199f383442 retrieve a feed for the given URL, store entries as feed_entry rows into the database
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 2
diff changeset
15 allFeeds = session.query(Feed)
e0199f383442 retrieve a feed for the given URL, store entries as feed_entry rows into the database
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 2
diff changeset
16 for feed in allFeeds:
11
e87c54b3a216 use the logging framework for printing messages
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 6
diff changeset
17 logger.info("feed: " + feed.name)
4
e0199f383442 retrieve a feed for the given URL, store entries as feed_entry rows into the database
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 2
diff changeset
18 for entry in feed.entries:
e0199f383442 retrieve a feed for the given URL, store entries as feed_entry rows into the database
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 2
diff changeset
19 print entry.title
e0199f383442 retrieve a feed for the given URL, store entries as feed_entry rows into the database
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 2
diff changeset
20
68
525a52169f60 Make FeedEntry table object available in Mapping to use it to expire old, read feed entries.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 64
diff changeset
21 def expireFeedEntries(session):
525a52169f60 Make FeedEntry table object available in Mapping to use it to expire old, read feed entries.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 64
diff changeset
22 expireDate = _calculateExpireDate()
525a52169f60 Make FeedEntry table object available in Mapping to use it to expire old, read feed entries.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 64
diff changeset
23 logger.info("expiring entries older than " + str(expireDate))
525a52169f60 Make FeedEntry table object available in Mapping to use it to expire old, read feed entries.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 64
diff changeset
24 feedEntry = Mapping.feedEntryTable
525a52169f60 Make FeedEntry table object available in Mapping to use it to expire old, read feed entries.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 64
diff changeset
25 deleteStatement = feedEntry.delete().where(
525a52169f60 Make FeedEntry table object available in Mapping to use it to expire old, read feed entries.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 64
diff changeset
26 and_(feedEntry.c.create_timestamp < expireDate, feedEntry.c.read == 1)
525a52169f60 Make FeedEntry table object available in Mapping to use it to expire old, read feed entries.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 64
diff changeset
27 )
525a52169f60 Make FeedEntry table object available in Mapping to use it to expire old, read feed entries.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 64
diff changeset
28 deleteStatement.execute()
525a52169f60 Make FeedEntry table object available in Mapping to use it to expire old, read feed entries.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 64
diff changeset
29
525a52169f60 Make FeedEntry table object available in Mapping to use it to expire old, read feed entries.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 64
diff changeset
30 def _calculateExpireDate():
525a52169f60 Make FeedEntry table object available in Mapping to use it to expire old, read feed entries.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 64
diff changeset
31 now = datetime.now()
525a52169f60 Make FeedEntry table object available in Mapping to use it to expire old, read feed entries.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 64
diff changeset
32 delta = timedelta(days=30)
525a52169f60 Make FeedEntry table object available in Mapping to use it to expire old, read feed entries.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 64
diff changeset
33 return now - delta
525a52169f60 Make FeedEntry table object available in Mapping to use it to expire old, read feed entries.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 64
diff changeset
34
2
8a624ee48a74 First skeleton for sqlalchemy: define the mapping and create the first feed
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
35 if __name__ == "__main__":
34
5813e3c10f14 move the database logic out into its own module. Make everything reload safe so that multiple sessions can be created from interactive sessions
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 14
diff changeset
36 util.configureLogging()
5813e3c10f14 move the database logic out into its own module. Make everything reload safe so that multiple sessions can be created from interactive sessions
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 14
diff changeset
37 session = Database.createSession()
2
8a624ee48a74 First skeleton for sqlalchemy: define the mapping and create the first feed
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
38
35
aaec263f07ca Feeds manage the point in time when the next update should happen. FeedUpdater only updates feeds that are due.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 34
diff changeset
39 #util.loadFeeds(session)
aaec263f07ca Feeds manage the point in time when the next update should happen. FeedUpdater only updates feeds that are due.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 34
diff changeset
40 #util.forceUpdateAllFeeds(session)
6
87317ba41816 add a creation date for the feed entry
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 5
diff changeset
41 #listFeeds(session)
64
d21f5025034d feedupdate-main updates all feeds per default
dirk@xanthippe.ping.de
parents: 35
diff changeset
42 #OpmlImport.createFeedsFromOPML(session, "feed-data/feeds.opml.xml")
69
d20e99d46d78 grr, need to re-enable the actual updating of entries
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 68
diff changeset
43 FeedUpdater.updateAllFeeds(session)
68
525a52169f60 Make FeedEntry table object available in Mapping to use it to expire old, read feed entries.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 64
diff changeset
44 expireFeedEntries(session)
35
aaec263f07ca Feeds manage the point in time when the next update should happen. FeedUpdater only updates feeds that are due.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 34
diff changeset
45
aaec263f07ca Feeds manage the point in time when the next update should happen. FeedUpdater only updates feeds that are due.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 34
diff changeset
46 session.close()