Mercurial > hg > Feedworm

annotate backend/sqlalchemy/FeedUpdater.py @ 197:e604c32f67aa

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
normalize the published date if the feed contains none
author dirk
date Tue, 24 Jan 2012 10:08:45 +0100
parents f4708d38419c
children bb3c851b18b1
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
4
e0199f383442 retrieve a feed for the given URL, store entries as feed_entry rows into the database
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
1
167
a3c945ce434c adjust the sqlalchemy backend to the changes in AbstractFeedUpdater
dirk
parents: 160
diff changeset
2 from Feed import Feed
a3c945ce434c adjust the sqlalchemy backend to the changes in AbstractFeedUpdater
dirk
parents: 160
diff changeset
3 from FeedEntry import FeedEntry
141
6ea813cfac33 pull out common code for updating a feed into an abstract class, have the sqlalchemy backend use that class.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 123
diff changeset
4 from backend.AbstractFeedUpdater import AbstractFeedUpdater, FeedUpdateException
5
bfd47f55d85b add the updated date of the feed
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 4
diff changeset
5 from datetime import datetime
11
e87c54b3a216 use the logging framework for printing messages
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 10
diff changeset
6 import logging
4
e0199f383442 retrieve a feed for the given URL, store entries as feed_entry rows into the database
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
7
28
72dfae865899 better logging when updating feeds, handle entries that have no id
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 27
diff changeset
8 log = logging.getLogger("FeedUpdater")
9
fd4c8bfa62d6 FeedUpdater throws an exception if the URL could not be retrieved successfully. Includes unit tests.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 7
diff changeset
9
168
f4708d38419c move methods around so the SqlAlchemyBackend only needs to import the class
dirk
parents: 167
diff changeset
10 def _findFeedsToUpdate(session):
35
aaec263f07ca Feeds manage the point in time when the next update should happen. FeedUpdater only updates feeds that are due.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 28
diff changeset
11 return session.query(Feed).filter(Feed.next_update < datetime.now())
aaec263f07ca Feeds manage the point in time when the next update should happen. FeedUpdater only updates feeds that are due.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 28
diff changeset
12
168
f4708d38419c move methods around so the SqlAlchemyBackend only needs to import the class
dirk
parents: 167
diff changeset
13 class FeedUpdater(AbstractFeedUpdater):
f4708d38419c move methods around so the SqlAlchemyBackend only needs to import the class
dirk
parents: 167
diff changeset
14 @staticmethod
f4708d38419c move methods around so the SqlAlchemyBackend only needs to import the class
dirk
parents: 167
diff changeset
15 def updateAllFeeds(preferences, session):
f4708d38419c move methods around so the SqlAlchemyBackend only needs to import the class
dirk
parents: 167
diff changeset
16 allFeeds = _findFeedsToUpdate(session)
f4708d38419c move methods around so the SqlAlchemyBackend only needs to import the class
dirk
parents: 167
diff changeset
17 for feed in allFeeds:
f4708d38419c move methods around so the SqlAlchemyBackend only needs to import the class
dirk
parents: 167
diff changeset
18 try:
f4708d38419c move methods around so the SqlAlchemyBackend only needs to import the class
dirk
parents: 167
diff changeset
19 FeedUpdater(preferences, session).update(feed)
f4708d38419c move methods around so the SqlAlchemyBackend only needs to import the class
dirk
parents: 167
diff changeset
20 except FeedUpdateException, fue:
f4708d38419c move methods around so the SqlAlchemyBackend only needs to import the class
dirk
parents: 167
diff changeset
21 log.warn("problems while updating feed " + feed.rss_url + ": " + str(fue))
f4708d38419c move methods around so the SqlAlchemyBackend only needs to import the class
dirk
parents: 167
diff changeset
22 session.commit()
123
862760b161b4 restructured adding a feed so that only the URL is passed into the backend - the rest of the operation is backend-internal
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 121
diff changeset
23
167
a3c945ce434c adjust the sqlalchemy backend to the changes in AbstractFeedUpdater
dirk
parents: 160
diff changeset
24 def __init__(self, preferences, session):
a3c945ce434c adjust the sqlalchemy backend to the changes in AbstractFeedUpdater
dirk
parents: 160
diff changeset
25 AbstractFeedUpdater.__init__(self, preferences)
4
e0199f383442 retrieve a feed for the given URL, store entries as feed_entry rows into the database
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
26 self.session = session
e0199f383442 retrieve a feed for the given URL, store entries as feed_entry rows into the database
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
27
141
6ea813cfac33 pull out common code for updating a feed into an abstract class, have the sqlalchemy backend use that class.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 123
diff changeset
28 def _processEntry(self, entry):
4
e0199f383442 retrieve a feed for the given URL, store entries as feed_entry rows into the database
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
29 feedEntry = FeedEntry.findById(entry.id, self.session)
e0199f383442 retrieve a feed for the given URL, store entries as feed_entry rows into the database
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
30 if feedEntry is None:
141
6ea813cfac33 pull out common code for updating a feed into an abstract class, have the sqlalchemy backend use that class.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 123
diff changeset
31 self._createFeedEntry(entry)
100
99807963d9e0 use the URL as feed title if the feed itself does not come with a title
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 85
diff changeset
32
141
6ea813cfac33 pull out common code for updating a feed into an abstract class, have the sqlalchemy backend use that class.
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 123
diff changeset
33 def _createFeedEntry(self, entry):
62
abc0516a1c0c FeedEntry provides a static method for creating new entries: better modularization and support for working with the class in interactive mode. FeedUpdater's normalize method is a module function now, again for ease of use in interactive scenarios
dirk@xanthippe.ping.de
parents: 58
diff changeset
34 new = FeedEntry.create(entry)
5
bfd47f55d85b add the updated date of the feed
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 4
diff changeset
35 new.feed = self.feed
bfd47f55d85b add the updated date of the feed
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 4
diff changeset
36 self.session.add(new)
66
97c4e94f99cf log when creating a new FeedEntry
dirk@xanthippe.ping.de
parents: 62
diff changeset
37 log.info("new feed entry: " + entry.title)
144
74217db92993 updating feeds on the couchdb backend works now
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 143
diff changeset
38
74217db92993 updating feeds on the couchdb backend works now
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 143
diff changeset
39 def _incrementFeedUpdateDate(self):
74217db92993 updating feeds on the couchdb backend works now
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 143
diff changeset
40 self.feed.incrementNextUpdateDate()