Mercurial > hg > Feedworm

annotate feedupdate-main.py @ 97:11036eabbc46

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
use a smaller default row height
author Dirk Olmes <dirk@xanthippe.ping.de>
date Tue, 28 Dec 2010 03:56:45 +0100
parents d20e99d46d78
children e50d446f9942 bf22b32ae9f7
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
2
8a624ee48a74 First skeleton for sqlalchemy: define the mapping and create the first feed
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
1 #!/usr/bin/env python
8a624ee48a74 First skeleton for sqlalchemy: define the mapping and create the first feed
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
2
34
5813e3c10f14 move the database logic out into its own module. Make everything reload safe so that multiple sessions can be created from interactive sessions
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 14
diff changeset
3 import Database
68
525a52169f60 Make FeedEntry table object available in Mapping to use it to expire old, read feed entries.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 64
diff changeset
4 from datetime import datetime, timedelta
2
8a624ee48a74 First skeleton for sqlalchemy: define the mapping and create the first feed
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
5 from Feed import Feed
35
aaec263f07ca Feeds manage the point in time when the next update should happen. FeedUpdater only updates feeds that are due.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 34
diff changeset
6 import FeedUpdater
11
e87c54b3a216 use the logging framework for printing messages
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 6
diff changeset
7 import logging
68
525a52169f60 Make FeedEntry table object available in Mapping to use it to expire old, read feed entries.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 64
diff changeset
8 import Mapping
525a52169f60 Make FeedEntry table object available in Mapping to use it to expire old, read feed entries.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 64
diff changeset
9 from sqlalchemy.sql import and_
34
5813e3c10f14 move the database logic out into its own module. Make everything reload safe so that multiple sessions can be created from interactive sessions
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 14
diff changeset
10 import util
2
8a624ee48a74 First skeleton for sqlalchemy: define the mapping and create the first feed
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
11
11
e87c54b3a216 use the logging framework for printing messages
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 6
diff changeset
12 logger = logging.getLogger("feedupdater")
e87c54b3a216 use the logging framework for printing messages
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 6
diff changeset
13
4
e0199f383442 retrieve a feed for the given URL, store entries as feed_entry rows into the database
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 2
diff changeset
14 def listFeeds(session):
e0199f383442 retrieve a feed for the given URL, store entries as feed_entry rows into the database
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 2
diff changeset
15 allFeeds = session.query(Feed)
e0199f383442 retrieve a feed for the given URL, store entries as feed_entry rows into the database
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 2
diff changeset
16 for feed in allFeeds:
11
e87c54b3a216 use the logging framework for printing messages
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 6
diff changeset
17 logger.info("feed: " + feed.name)
4
e0199f383442 retrieve a feed for the given URL, store entries as feed_entry rows into the database
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 2
diff changeset
18 for entry in feed.entries:
e0199f383442 retrieve a feed for the given URL, store entries as feed_entry rows into the database
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 2
diff changeset
19 print entry.title
e0199f383442 retrieve a feed for the given URL, store entries as feed_entry rows into the database
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 2
diff changeset
20
68
525a52169f60 Make FeedEntry table object available in Mapping to use it to expire old, read feed entries.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 64
diff changeset
21 def expireFeedEntries(session):
525a52169f60 Make FeedEntry table object available in Mapping to use it to expire old, read feed entries.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 64
diff changeset
22 expireDate = _calculateExpireDate()
525a52169f60 Make FeedEntry table object available in Mapping to use it to expire old, read feed entries.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 64
diff changeset
23 logger.info("expiring entries older than " + str(expireDate))
525a52169f60 Make FeedEntry table object available in Mapping to use it to expire old, read feed entries.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 64
diff changeset
24 feedEntry = Mapping.feedEntryTable
525a52169f60 Make FeedEntry table object available in Mapping to use it to expire old, read feed entries.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 64
diff changeset
25 deleteStatement = feedEntry.delete().where(
525a52169f60 Make FeedEntry table object available in Mapping to use it to expire old, read feed entries.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 64
diff changeset
26 and_(feedEntry.c.create_timestamp < expireDate, feedEntry.c.read == 1)
525a52169f60 Make FeedEntry table object available in Mapping to use it to expire old, read feed entries.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 64
diff changeset
27 )
525a52169f60 Make FeedEntry table object available in Mapping to use it to expire old, read feed entries.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 64
diff changeset
28 deleteStatement.execute()
525a52169f60 Make FeedEntry table object available in Mapping to use it to expire old, read feed entries.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 64
diff changeset
29
525a52169f60 Make FeedEntry table object available in Mapping to use it to expire old, read feed entries.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 64
diff changeset
30 def _calculateExpireDate():
525a52169f60 Make FeedEntry table object available in Mapping to use it to expire old, read feed entries.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 64
diff changeset
31 now = datetime.now()
525a52169f60 Make FeedEntry table object available in Mapping to use it to expire old, read feed entries.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 64
diff changeset
32 delta = timedelta(days=30)
525a52169f60 Make FeedEntry table object available in Mapping to use it to expire old, read feed entries.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 64
diff changeset
33 return now - delta
525a52169f60 Make FeedEntry table object available in Mapping to use it to expire old, read feed entries.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 64
diff changeset
34
2
8a624ee48a74 First skeleton for sqlalchemy: define the mapping and create the first feed
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
35 if __name__ == "__main__":
34
5813e3c10f14 move the database logic out into its own module. Make everything reload safe so that multiple sessions can be created from interactive sessions
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 14
diff changeset
36 util.configureLogging()
5813e3c10f14 move the database logic out into its own module. Make everything reload safe so that multiple sessions can be created from interactive sessions
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 14
diff changeset
37 session = Database.createSession()
2
8a624ee48a74 First skeleton for sqlalchemy: define the mapping and create the first feed
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
38
35
aaec263f07ca Feeds manage the point in time when the next update should happen. FeedUpdater only updates feeds that are due.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 34
diff changeset
39 #util.loadFeeds(session)
aaec263f07ca Feeds manage the point in time when the next update should happen. FeedUpdater only updates feeds that are due.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 34
diff changeset
40 #util.forceUpdateAllFeeds(session)
6
87317ba41816 add a creation date for the feed entry
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 5
diff changeset
41 #listFeeds(session)
64
d21f5025034d feedupdate-main updates all feeds per default
dirk@xanthippe.ping.de
parents: 35
diff changeset
42 #OpmlImport.createFeedsFromOPML(session, "feed-data/feeds.opml.xml")
69
d20e99d46d78 grr, need to re-enable the actual updating of entries
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 68
diff changeset
43 FeedUpdater.updateAllFeeds(session)
68
525a52169f60 Make FeedEntry table object available in Mapping to use it to expire old, read feed entries.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 64
diff changeset
44 expireFeedEntries(session)
35
aaec263f07ca Feeds manage the point in time when the next update should happen. FeedUpdater only updates feeds that are due.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 34
diff changeset
45
aaec263f07ca Feeds manage the point in time when the next update should happen. FeedUpdater only updates feeds that are due.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 34
diff changeset
46 session.close()