Mercurial > hg > Feedworm
annotate backend/couchdb/CouchDbBackend.py @ 255:b4c83e9b9c7a
migration from couchdb to arangodb
author | Dirk Olmes <dirk@xanthippe.ping.de> |
---|---|
date | Thu, 29 Nov 2018 18:46:21 +0100 |
parents | 8e73a8ae863f |
children | f79be01821c4 |
rev | line source |
---|---|
217
bb3c851b18b1
add source file endcoding header
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
214
diff
changeset
|
1 # -*- coding: utf-8 -*- |
218
699d8f1cebd4
unify imports, especially Qt imports. Use consistent super syntax
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
217
diff
changeset
|
2 import couchdb |
699d8f1cebd4
unify imports, especially Qt imports. Use consistent super syntax
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
217
diff
changeset
|
3 import logging |
144
74217db92993
updating feeds on the couchdb backend works now
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
139
diff
changeset
|
4 from FeedUpdater import FeedUpdater |
133
9e1e6b96d8b0
implement proxyHost/proxyPort in Preferences
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
132
diff
changeset
|
5 from Preferences import Preferences |
155
a05719a6175e
move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
148
diff
changeset
|
6 from backend.AbstractBackend import AbstractBackend |
174 | 7 from backend.couchdb import CouchDb |
139
2cd30af937fa
add the required methods for determining if a feed needs to be updated
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
133
diff
changeset
|
8 from backend.couchdb.Feed import Feed |
146
8ec20377bcb0
move getting the entries for a feed to the backend so that the couchdb backend can use a custom view for feed entries
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
144
diff
changeset
|
9 from backend.couchdb.FeedEntry import FeedEntry |
133
9e1e6b96d8b0
implement proxyHost/proxyPort in Preferences
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
132
diff
changeset
|
10 |
245 | 11 """ |
12 Backend that uses CouchDB for persistence | |
13 """ | |
155
a05719a6175e
move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
148
diff
changeset
|
14 class CouchDbBackend(AbstractBackend): |
132
63d6d60d37ff
new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff
changeset
|
15 |
63d6d60d37ff
new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff
changeset
|
16 def __init__(self): |
218
699d8f1cebd4
unify imports, especially Qt imports. Use consistent super syntax
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
217
diff
changeset
|
17 super(CouchDbBackend, self).__init__() |
169
91a24f499318
introdue an abstraction for the name of the database so it can be changed via commandline parameter
dirk
parents:
166
diff
changeset
|
18 CouchDb.init() |
180
a4832a180c69
allow setting the URL to the database via command line
dirk
parents:
176
diff
changeset
|
19 server = self._initServer() |
169
91a24f499318
introdue an abstraction for the name of the database so it can be changed via commandline parameter
dirk
parents:
166
diff
changeset
|
20 self.database = server[CouchDb.database] |
158
e8bb107a74e1
all preferences are stored in a single JSON document in the couchdb backend. PreferencesDialog converts QString to a proper Python datatybe before sending it to the backend.
dirk
parents:
155
diff
changeset
|
21 self.prefs = None |
132
63d6d60d37ff
new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff
changeset
|
22 |
180
a4832a180c69
allow setting the URL to the database via command line
dirk
parents:
176
diff
changeset
|
23 def _initServer(self): |
a4832a180c69
allow setting the URL to the database via command line
dirk
parents:
176
diff
changeset
|
24 if CouchDb.database_url is not None: |
a4832a180c69
allow setting the URL to the database via command line
dirk
parents:
176
diff
changeset
|
25 return couchdb.Server(CouchDb.database_url) |
a4832a180c69
allow setting the URL to the database via command line
dirk
parents:
176
diff
changeset
|
26 else: |
a4832a180c69
allow setting the URL to the database via command line
dirk
parents:
176
diff
changeset
|
27 return couchdb.Server() |
a4832a180c69
allow setting the URL to the database via command line
dirk
parents:
176
diff
changeset
|
28 |
132
63d6d60d37ff
new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff
changeset
|
29 def preferences(self): |
158
e8bb107a74e1
all preferences are stored in a single JSON document in the couchdb backend. PreferencesDialog converts QString to a proper Python datatybe before sending it to the backend.
dirk
parents:
155
diff
changeset
|
30 if self.prefs is None: |
e8bb107a74e1
all preferences are stored in a single JSON document in the couchdb backend. PreferencesDialog converts QString to a proper Python datatybe before sending it to the backend.
dirk
parents:
155
diff
changeset
|
31 self.prefs = Preferences(self.database) |
e8bb107a74e1
all preferences are stored in a single JSON document in the couchdb backend. PreferencesDialog converts QString to a proper Python datatybe before sending it to the backend.
dirk
parents:
155
diff
changeset
|
32 return self.prefs |
132
63d6d60d37ff
new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff
changeset
|
33 |
155
a05719a6175e
move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
148
diff
changeset
|
34 # |
a05719a6175e
move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
148
diff
changeset
|
35 # handling of feeds |
a05719a6175e
move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
148
diff
changeset
|
36 # |
a05719a6175e
move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
148
diff
changeset
|
37 |
132
63d6d60d37ff
new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff
changeset
|
38 def getFeeds(self): |
146
8ec20377bcb0
move getting the entries for a feed to the backend so that the couchdb backend can use a custom view for feed entries
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
144
diff
changeset
|
39 if self.preferences().showOnlyUnreadFeeds(): |
155
a05719a6175e
move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
148
diff
changeset
|
40 self.feeds = self._getUnreadFeeds() |
146
8ec20377bcb0
move getting the entries for a feed to the backend so that the couchdb backend can use a custom view for feed entries
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
144
diff
changeset
|
41 else: |
148
c5a427d46703
displaying entries for a feed works now with the couchdb backend
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
146
diff
changeset
|
42 # make sure that the results are actually fetched into memory, otherwise we'll pass |
c5a427d46703
displaying entries for a feed works now with the couchdb backend
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
146
diff
changeset
|
43 # a ViewResults instance around which is not what we want |
155
a05719a6175e
move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
148
diff
changeset
|
44 self.feeds = list(Feed.all(self.database)) |
a05719a6175e
move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
148
diff
changeset
|
45 return self.feeds |
146
8ec20377bcb0
move getting the entries for a feed to the backend so that the couchdb backend can use a custom view for feed entries
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
144
diff
changeset
|
46 |
8ec20377bcb0
move getting the entries for a feed to the backend so that the couchdb backend can use a custom view for feed entries
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
144
diff
changeset
|
47 def _getUnreadFeeds(self): |
175
57e324fa4350
implement getting a list of feeds that have unread entries
dirk
parents:
174
diff
changeset
|
48 viewResults = self.database.view(CouchDb.feedsWithUnreadEntries(), group=True) |
176 | 49 feedsWithUnreadEntries = [] |
175
57e324fa4350
implement getting a list of feeds that have unread entries
dirk
parents:
174
diff
changeset
|
50 for row in viewResults: |
176 | 51 feed = Feed.load(self.database, row["key"]) |
236
c5250fcc3881
remove a workaround along with its wrong explanation
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
233
diff
changeset
|
52 feedsWithUnreadEntries.append(feed) |
176 | 53 return feedsWithUnreadEntries |
132
63d6d60d37ff
new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff
changeset
|
54 |
155
a05719a6175e
move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
148
diff
changeset
|
55 def _retrieveEntriesForSelectedFeed(self, hideReadEntries): |
183 | 56 viewResults = FeedEntry.entriesForFeed(self.selectedFeed, self.database) |
182 | 57 if hideReadEntries: |
233
e34c53a3e407
fixes from eric's style check
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
218
diff
changeset
|
58 filterFunc = lambda feedEntry: feedEntry.read is False |
183 | 59 viewResults = filter(filterFunc, viewResults) |
181 | 60 return viewResults |
155
a05719a6175e
move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
148
diff
changeset
|
61 |
a05719a6175e
move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
148
diff
changeset
|
62 def markSelectedFeedAsRead(self): |
a05719a6175e
move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
148
diff
changeset
|
63 for feedEntry in self.entriesForSelectedFeed(): |
a05719a6175e
move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
148
diff
changeset
|
64 feedEntry.markRead(self.database) |
a05719a6175e
move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
148
diff
changeset
|
65 |
a05719a6175e
move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
148
diff
changeset
|
66 # |
a05719a6175e
move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
148
diff
changeset
|
67 # handling of the selected feed entry |
a05719a6175e
move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
148
diff
changeset
|
68 # |
a05719a6175e
move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
148
diff
changeset
|
69 |
a05719a6175e
move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
148
diff
changeset
|
70 def _markSelectedFeedEntryRead(self): |
a05719a6175e
move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
148
diff
changeset
|
71 self.selectedFeedEntry.markRead(self.database) |
a05719a6175e
move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
148
diff
changeset
|
72 |
a05719a6175e
move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
148
diff
changeset
|
73 def toggleSelectedFeedEntryRead(self): |
a05719a6175e
move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
148
diff
changeset
|
74 self.selectedFeedEntry.toggleRead(self.database) |
a05719a6175e
move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
148
diff
changeset
|
75 |
132
63d6d60d37ff
new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff
changeset
|
76 def createFeed(self, url): |
166
04c3b9796b89
feedparser uses the proxy now if one is configured. To implement this the FeedUpdater had to change a bit - sqlalchemy backend is not yet refactored.
dirk
parents:
164
diff
changeset
|
77 feed = Feed.create(url) |
04c3b9796b89
feedparser uses the proxy now if one is configured. To implement this the FeedUpdater had to change a bit - sqlalchemy backend is not yet refactored.
dirk
parents:
164
diff
changeset
|
78 feed.store(self.database) |
04c3b9796b89
feedparser uses the proxy now if one is configured. To implement this the FeedUpdater had to change a bit - sqlalchemy backend is not yet refactored.
dirk
parents:
164
diff
changeset
|
79 FeedUpdater(self.database, self.preferences()).update(feed) |
132
63d6d60d37ff
new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff
changeset
|
80 |
63d6d60d37ff
new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff
changeset
|
81 def updateFeed(self, feed, changes): |
162 | 82 for key in changes.keys(): |
83 feed[key] = changes[key] | |
84 feed.store(self.database) | |
132
63d6d60d37ff
new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff
changeset
|
85 |
164 | 86 def deleteSelectedFeed(self): |
240
1b98925facf6
Bugfix: use the existing _retrieveEntriesForSelectedFeed method to retrieve all articles when deleting a feed
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
236
diff
changeset
|
87 viewResults = self._retrieveEntriesForSelectedFeed(False) |
164 | 88 for row in viewResults: |
89 del self.database[row.id] | |
90 del self.database[self.selectedFeed.id] | |
132
63d6d60d37ff
new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff
changeset
|
91 |
146
8ec20377bcb0
move getting the entries for a feed to the backend so that the couchdb backend can use a custom view for feed entries
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
144
diff
changeset
|
92 def entriesForFeed(self, feed, hideReadEntries): |
169
91a24f499318
introdue an abstraction for the name of the database so it can be changed via commandline parameter
dirk
parents:
166
diff
changeset
|
93 viewName = CouchDb.feedEntriesByFeed() |
146
8ec20377bcb0
move getting the entries for a feed to the backend so that the couchdb backend can use a custom view for feed entries
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
144
diff
changeset
|
94 if hideReadEntries: |
169
91a24f499318
introdue an abstraction for the name of the database so it can be changed via commandline parameter
dirk
parents:
166
diff
changeset
|
95 viewName = CouchDb.unreadFeedEntriesByFeed() |
148
c5a427d46703
displaying entries for a feed works now with the couchdb backend
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
146
diff
changeset
|
96 return list(FeedEntry.view(self.database, viewName)) |
146
8ec20377bcb0
move getting the entries for a feed to the backend so that the couchdb backend can use a custom view for feed entries
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
144
diff
changeset
|
97 |
155
a05719a6175e
move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
148
diff
changeset
|
98 def markFeedEntriesAsRead(self, indices): |
a05719a6175e
move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
148
diff
changeset
|
99 for index in indices: |
a05719a6175e
move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
148
diff
changeset
|
100 feedEntry = self.entriesForSelectedFeed()[index] |
a05719a6175e
move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
148
diff
changeset
|
101 feedEntry.markRead(self.database) |
132
63d6d60d37ff
new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff
changeset
|
102 |
63d6d60d37ff
new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff
changeset
|
103 def updateAllFeeds(self): |
213
524cbf9e413c
use correct TODO tags so they show up in the tasks view in Eclipse
dirk
parents:
204
diff
changeset
|
104 # TODO: use a view instead of iterating all feeds |
139
2cd30af937fa
add the required methods for determining if a feed needs to be updated
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
133
diff
changeset
|
105 allFeeds = Feed.all(self.database) |
2cd30af937fa
add the required methods for determining if a feed needs to be updated
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
133
diff
changeset
|
106 for feed in allFeeds: |
2cd30af937fa
add the required methods for determining if a feed needs to be updated
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
133
diff
changeset
|
107 if feed.needsUpdate(): |
211
1ac0b8e2feae
wrap the individual feed update with an exception handler so that a sinlge broken feed doesn't halt the entire update process
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
204
diff
changeset
|
108 try: |
1ac0b8e2feae
wrap the individual feed update with an exception handler so that a sinlge broken feed doesn't halt the entire update process
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
204
diff
changeset
|
109 FeedUpdater(self.database, self.preferences()).update(feed) |
1ac0b8e2feae
wrap the individual feed update with an exception handler so that a sinlge broken feed doesn't halt the entire update process
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
204
diff
changeset
|
110 except Exception as ex: |
1ac0b8e2feae
wrap the individual feed update with an exception handler so that a sinlge broken feed doesn't halt the entire update process
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
204
diff
changeset
|
111 logging.getLogger("FeedUpdate").error("Exception during fetch: " + str(ex)) |
132
63d6d60d37ff
new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff
changeset
|
112 |
63d6d60d37ff
new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff
changeset
|
113 def expireFeedEntries(self): |
201 | 114 expireDate = self._calculateExpireDate() |
174 | 115 logger = logging.getLogger("expiry") |
116 logger.info("expiring entries older than " + str(expireDate)) | |
117 for entry in FeedEntry.getReadFeedEntriesOlderThan(expireDate, self.database): | |
118 del self.database[entry.id] |