Mercurial > hg > Feedworm

annotate backend/couchdb/CouchDbBackend.py @ 197:e604c32f67aa

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
normalize the published date if the feed contains none
author dirk
date Tue, 24 Jan 2012 10:08:45 +0100
parents 7157bfd3f4d4
children 50b578588405
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
132
63d6d60d37ff new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
1
144
74217db92993 updating feeds on the couchdb backend works now
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 139
diff changeset
2 from FeedUpdater import FeedUpdater
133
9e1e6b96d8b0 implement proxyHost/proxyPort in Preferences
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 132
diff changeset
3 from Preferences import Preferences
155
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
4 from backend.AbstractBackend import AbstractBackend
174
d0ced79b5030 implement expiring read feed entries
dirk
parents: 169
diff changeset
5 from backend.couchdb import CouchDb
139
2cd30af937fa add the required methods for determining if a feed needs to be updated
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 133
diff changeset
6 from backend.couchdb.Feed import Feed
146
8ec20377bcb0 move getting the entries for a feed to the backend so that the couchdb backend can use a custom view for feed entries
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 144
diff changeset
7 from backend.couchdb.FeedEntry import FeedEntry
132
63d6d60d37ff new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
8 import couchdb
174
d0ced79b5030 implement expiring read feed entries
dirk
parents: 169
diff changeset
9 import logging
133
9e1e6b96d8b0 implement proxyHost/proxyPort in Preferences
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 132
diff changeset
10
155
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
11 class CouchDbBackend(AbstractBackend):
132
63d6d60d37ff new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
12 '''
63d6d60d37ff new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
13 Backend that uses CouchDB for persistence
63d6d60d37ff new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
14 '''
63d6d60d37ff new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
15
63d6d60d37ff new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
16 def __init__(self):
169
91a24f499318 introdue an abstraction for the name of the database so it can be changed via commandline parameter
dirk
parents: 166
diff changeset
17 CouchDb.init()
180
a4832a180c69 allow setting the URL to the database via command line
dirk
parents: 176
diff changeset
18 server = self._initServer()
169
91a24f499318 introdue an abstraction for the name of the database so it can be changed via commandline parameter
dirk
parents: 166
diff changeset
19 self.database = server[CouchDb.database]
158
e8bb107a74e1 all preferences are stored in a single JSON document in the couchdb backend. PreferencesDialog converts QString to a proper Python datatybe before sending it to the backend.
dirk
parents: 155
diff changeset
20 self.prefs = None
132
63d6d60d37ff new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
21
180
a4832a180c69 allow setting the URL to the database via command line
dirk
parents: 176
diff changeset
22 def _initServer(self):
a4832a180c69 allow setting the URL to the database via command line
dirk
parents: 176
diff changeset
23 if CouchDb.database_url is not None:
a4832a180c69 allow setting the URL to the database via command line
dirk
parents: 176
diff changeset
24 return couchdb.Server(CouchDb.database_url)
a4832a180c69 allow setting the URL to the database via command line
dirk
parents: 176
diff changeset
25 else:
a4832a180c69 allow setting the URL to the database via command line
dirk
parents: 176
diff changeset
26 return couchdb.Server()
a4832a180c69 allow setting the URL to the database via command line
dirk
parents: 176
diff changeset
27
132
63d6d60d37ff new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
28 def preferences(self):
158
e8bb107a74e1 all preferences are stored in a single JSON document in the couchdb backend. PreferencesDialog converts QString to a proper Python datatybe before sending it to the backend.
dirk
parents: 155
diff changeset
29 if self.prefs is None:
e8bb107a74e1 all preferences are stored in a single JSON document in the couchdb backend. PreferencesDialog converts QString to a proper Python datatybe before sending it to the backend.
dirk
parents: 155
diff changeset
30 self.prefs = Preferences(self.database)
e8bb107a74e1 all preferences are stored in a single JSON document in the couchdb backend. PreferencesDialog converts QString to a proper Python datatybe before sending it to the backend.
dirk
parents: 155
diff changeset
31 return self.prefs
132
63d6d60d37ff new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
32
155
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
33 #
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
34 # handling of feeds
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
35 #
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
36
132
63d6d60d37ff new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
37 def getFeeds(self):
146
8ec20377bcb0 move getting the entries for a feed to the backend so that the couchdb backend can use a custom view for feed entries
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 144
diff changeset
38 if self.preferences().showOnlyUnreadFeeds():
155
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
39 self.feeds = self._getUnreadFeeds()
146
8ec20377bcb0 move getting the entries for a feed to the backend so that the couchdb backend can use a custom view for feed entries
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 144
diff changeset
40 else:
148
c5a427d46703 displaying entries for a feed works now with the couchdb backend
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 146
diff changeset
41 # make sure that the results are actually fetched into memory, otherwise we'll pass
c5a427d46703 displaying entries for a feed works now with the couchdb backend
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 146
diff changeset
42 # a ViewResults instance around which is not what we want
155
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
43 self.feeds = list(Feed.all(self.database))
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
44 return self.feeds
146
8ec20377bcb0 move getting the entries for a feed to the backend so that the couchdb backend can use a custom view for feed entries
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 144
diff changeset
45
8ec20377bcb0 move getting the entries for a feed to the backend so that the couchdb backend can use a custom view for feed entries
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 144
diff changeset
46 def _getUnreadFeeds(self):
175
57e324fa4350 implement getting a list of feeds that have unread entries
dirk
parents: 174
diff changeset
47 viewResults = self.database.view(CouchDb.feedsWithUnreadEntries(), group=True)
176
7001070d0bd5 clean code
dirk
parents: 175
diff changeset
48 feedsWithUnreadEntries = []
175
57e324fa4350 implement getting a list of feeds that have unread entries
dirk
parents: 174
diff changeset
49 for row in viewResults:
176
7001070d0bd5 clean code
dirk
parents: 175
diff changeset
50 feed = Feed.load(self.database, row["key"])
7001070d0bd5 clean code
dirk
parents: 175
diff changeset
51 feedsWithUnreadEntries.append(feed)
7001070d0bd5 clean code
dirk
parents: 175
diff changeset
52 return feedsWithUnreadEntries
132
63d6d60d37ff new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
53
155
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
54 def _retrieveEntriesForSelectedFeed(self, hideReadEntries):
183
ac1de0a467fd clean up
dirk
parents: 182
diff changeset
55 viewResults = FeedEntry.entriesForFeed(self.selectedFeed, self.database)
182
964661153b97 hide read feed entries if configured
dirk
parents: 181
diff changeset
56 if hideReadEntries:
183
ac1de0a467fd clean up
dirk
parents: 182
diff changeset
57 filterFunc = lambda feedEntry: feedEntry.read == False
ac1de0a467fd clean up
dirk
parents: 182
diff changeset
58 viewResults = filter(filterFunc, viewResults)
181
1ab48428e2f9 sort the feed entries
dirk
parents: 180
diff changeset
59 return viewResults
155
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
60
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
61 def markSelectedFeedAsRead(self):
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
62 for feedEntry in self.entriesForSelectedFeed():
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
63 feedEntry.markRead(self.database)
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
64
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
65 #
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
66 # handling of the selected feed entry
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
67 #
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
68
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
69 def _markSelectedFeedEntryRead(self):
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
70 self.selectedFeedEntry.markRead(self.database)
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
71
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
72 def toggleSelectedFeedEntryRead(self):
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
73 self.selectedFeedEntry.toggleRead(self.database)
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
74
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
75
132
63d6d60d37ff new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
76 def createFeed(self, url):
166
04c3b9796b89 feedparser uses the proxy now if one is configured. To implement this the FeedUpdater had to change a bit - sqlalchemy backend is not yet refactored.
dirk
parents: 164
diff changeset
77 feed = Feed.create(url)
04c3b9796b89 feedparser uses the proxy now if one is configured. To implement this the FeedUpdater had to change a bit - sqlalchemy backend is not yet refactored.
dirk
parents: 164
diff changeset
78 feed.store(self.database)
04c3b9796b89 feedparser uses the proxy now if one is configured. To implement this the FeedUpdater had to change a bit - sqlalchemy backend is not yet refactored.
dirk
parents: 164
diff changeset
79 FeedUpdater(self.database, self.preferences()).update(feed)
132
63d6d60d37ff new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
80
63d6d60d37ff new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
81 def updateFeed(self, feed, changes):
162
ab2b26412b77 setting a feed's properties via the GUI works now
dirk
parents: 161
diff changeset
82 for key in changes.keys():
ab2b26412b77 setting a feed's properties via the GUI works now
dirk
parents: 161
diff changeset
83 feed[key] = changes[key]
ab2b26412b77 setting a feed's properties via the GUI works now
dirk
parents: 161
diff changeset
84 feed.store(self.database)
132
63d6d60d37ff new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
85
164
3eeda7cec39b delete a feed via the GUI works now
dirk
parents: 162
diff changeset
86 def deleteSelectedFeed(self):
169
91a24f499318 introdue an abstraction for the name of the database so it can be changed via commandline parameter
dirk
parents: 166
diff changeset
87 viewResults = self.database.view(CouchDb.feedEntriesByFeed(), key=self.selectedFeed.id)
164
3eeda7cec39b delete a feed via the GUI works now
dirk
parents: 162
diff changeset
88 for row in viewResults:
3eeda7cec39b delete a feed via the GUI works now
dirk
parents: 162
diff changeset
89 del self.database[row.id]
3eeda7cec39b delete a feed via the GUI works now
dirk
parents: 162
diff changeset
90 del self.database[self.selectedFeed.id]
132
63d6d60d37ff new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
91
146
8ec20377bcb0 move getting the entries for a feed to the backend so that the couchdb backend can use a custom view for feed entries
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 144
diff changeset
92 def entriesForFeed(self, feed, hideReadEntries):
169
91a24f499318 introdue an abstraction for the name of the database so it can be changed via commandline parameter
dirk
parents: 166
diff changeset
93 viewName = CouchDb.feedEntriesByFeed()
146
8ec20377bcb0 move getting the entries for a feed to the backend so that the couchdb backend can use a custom view for feed entries
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 144
diff changeset
94 if hideReadEntries:
169
91a24f499318 introdue an abstraction for the name of the database so it can be changed via commandline parameter
dirk
parents: 166
diff changeset
95 viewName = CouchDb.unreadFeedEntriesByFeed()
148
c5a427d46703 displaying entries for a feed works now with the couchdb backend
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 146
diff changeset
96 return list(FeedEntry.view(self.database, viewName))
146
8ec20377bcb0 move getting the entries for a feed to the backend so that the couchdb backend can use a custom view for feed entries
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 144
diff changeset
97
155
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
98 def markFeedEntriesAsRead(self, indices):
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
99 for index in indices:
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
100 feedEntry = self.entriesForSelectedFeed()[index]
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
101 feedEntry.markRead(self.database)
132
63d6d60d37ff new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
102
63d6d60d37ff new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
103 def updateAllFeeds(self):
139
2cd30af937fa add the required methods for determining if a feed needs to be updated
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 133
diff changeset
104 # TODO use a view instead of iterating all feeds
2cd30af937fa add the required methods for determining if a feed needs to be updated
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 133
diff changeset
105 allFeeds = Feed.all(self.database)
2cd30af937fa add the required methods for determining if a feed needs to be updated
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 133
diff changeset
106 for feed in allFeeds:
2cd30af937fa add the required methods for determining if a feed needs to be updated
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 133
diff changeset
107 if feed.needsUpdate():
166
04c3b9796b89 feedparser uses the proxy now if one is configured. To implement this the FeedUpdater had to change a bit - sqlalchemy backend is not yet refactored.
dirk
parents: 164
diff changeset
108 FeedUpdater(self.database, self.preferences()).update(feed)
132
63d6d60d37ff new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
109
63d6d60d37ff new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
110 def expireFeedEntries(self):
174
d0ced79b5030 implement expiring read feed entries
dirk
parents: 169
diff changeset
111 logger = logging.getLogger("expiry")
d0ced79b5030 implement expiring read feed entries
dirk
parents: 169
diff changeset
112 expireDate = self._calculateExpireDate()
d0ced79b5030 implement expiring read feed entries
dirk
parents: 169
diff changeset
113 logger.info("expiring entries older than " + str(expireDate))
d0ced79b5030 implement expiring read feed entries
dirk
parents: 169
diff changeset
114 for entry in FeedEntry.getReadFeedEntriesOlderThan(expireDate, self.database):
d0ced79b5030 implement expiring read feed entries
dirk
parents: 169
diff changeset
115 del self.database[entry.id]