Mercurial > hg > Feedworm

annotate backend/couchdb/CouchDbBackend.py @ 259:304917762618 default tip

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
implementation of feed updates
author Dirk Olmes <dirk@xanthippe.ping.de>
date Tue, 12 Mar 2019 02:41:22 +0100
parents f79be01821c4
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
217
bb3c851b18b1 add source file endcoding header
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 214
diff changeset
1 # -*- coding: utf-8 -*-
218
699d8f1cebd4 unify imports, especially Qt imports. Use consistent super syntax
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 217
diff changeset
2 import couchdb
699d8f1cebd4 unify imports, especially Qt imports. Use consistent super syntax
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 217
diff changeset
3 import logging
144
74217db92993 updating feeds on the couchdb backend works now
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 139
diff changeset
4 from FeedUpdater import FeedUpdater
133
9e1e6b96d8b0 implement proxyHost/proxyPort in Preferences
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 132
diff changeset
5 from Preferences import Preferences
155
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
6 from backend.AbstractBackend import AbstractBackend
174
d0ced79b5030 implement expiring read feed entries
dirk
parents: 169
diff changeset
7 from backend.couchdb import CouchDb
139
2cd30af937fa add the required methods for determining if a feed needs to be updated
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 133
diff changeset
8 from backend.couchdb.Feed import Feed
146
8ec20377bcb0 move getting the entries for a feed to the backend so that the couchdb backend can use a custom view for feed entries
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 144
diff changeset
9 from backend.couchdb.FeedEntry import FeedEntry
133
9e1e6b96d8b0 implement proxyHost/proxyPort in Preferences
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 132
diff changeset
10
245
8e73a8ae863f Fix the docstrings
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 240
diff changeset
11 """
8e73a8ae863f Fix the docstrings
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 240
diff changeset
12 Backend that uses CouchDB for persistence
8e73a8ae863f Fix the docstrings
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 240
diff changeset
13 """
155
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
14 class CouchDbBackend(AbstractBackend):
132
63d6d60d37ff new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
15
63d6d60d37ff new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
16 def __init__(self):
218
699d8f1cebd4 unify imports, especially Qt imports. Use consistent super syntax
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 217
diff changeset
17 super(CouchDbBackend, self).__init__()
169
91a24f499318 introdue an abstraction for the name of the database so it can be changed via commandline parameter
dirk
parents: 166
diff changeset
18 CouchDb.init()
180
a4832a180c69 allow setting the URL to the database via command line
dirk
parents: 176
diff changeset
19 server = self._initServer()
169
91a24f499318 introdue an abstraction for the name of the database so it can be changed via commandline parameter
dirk
parents: 166
diff changeset
20 self.database = server[CouchDb.database]
158
e8bb107a74e1 all preferences are stored in a single JSON document in the couchdb backend. PreferencesDialog converts QString to a proper Python datatybe before sending it to the backend.
dirk
parents: 155
diff changeset
21 self.prefs = None
132
63d6d60d37ff new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
22
180
a4832a180c69 allow setting the URL to the database via command line
dirk
parents: 176
diff changeset
23 def _initServer(self):
a4832a180c69 allow setting the URL to the database via command line
dirk
parents: 176
diff changeset
24 if CouchDb.database_url is not None:
a4832a180c69 allow setting the URL to the database via command line
dirk
parents: 176
diff changeset
25 return couchdb.Server(CouchDb.database_url)
a4832a180c69 allow setting the URL to the database via command line
dirk
parents: 176
diff changeset
26 else:
a4832a180c69 allow setting the URL to the database via command line
dirk
parents: 176
diff changeset
27 return couchdb.Server()
a4832a180c69 allow setting the URL to the database via command line
dirk
parents: 176
diff changeset
28
132
63d6d60d37ff new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
29 def preferences(self):
158
e8bb107a74e1 all preferences are stored in a single JSON document in the couchdb backend. PreferencesDialog converts QString to a proper Python datatybe before sending it to the backend.
dirk
parents: 155
diff changeset
30 if self.prefs is None:
e8bb107a74e1 all preferences are stored in a single JSON document in the couchdb backend. PreferencesDialog converts QString to a proper Python datatybe before sending it to the backend.
dirk
parents: 155
diff changeset
31 self.prefs = Preferences(self.database)
e8bb107a74e1 all preferences are stored in a single JSON document in the couchdb backend. PreferencesDialog converts QString to a proper Python datatybe before sending it to the backend.
dirk
parents: 155
diff changeset
32 return self.prefs
132
63d6d60d37ff new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
33
155
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
34 #
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
35 # handling of feeds
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
36 #
256
f79be01821c4 Arangodb backend, first version which barely works for reading
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 245
diff changeset
37 def getUnreadFeeds(self):
175
57e324fa4350 implement getting a list of feeds that have unread entries
dirk
parents: 174
diff changeset
38 viewResults = self.database.view(CouchDb.feedsWithUnreadEntries(), group=True)
176
7001070d0bd5 clean code
dirk
parents: 175
diff changeset
39 feedsWithUnreadEntries = []
175
57e324fa4350 implement getting a list of feeds that have unread entries
dirk
parents: 174
diff changeset
40 for row in viewResults:
176
7001070d0bd5 clean code
dirk
parents: 175
diff changeset
41 feed = Feed.load(self.database, row["key"])
236
c5250fcc3881 remove a workaround along with its wrong explanation
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 233
diff changeset
42 feedsWithUnreadEntries.append(feed)
176
7001070d0bd5 clean code
dirk
parents: 175
diff changeset
43 return feedsWithUnreadEntries
132
63d6d60d37ff new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
44
256
f79be01821c4 Arangodb backend, first version which barely works for reading
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 245
diff changeset
45 def getAllFeeds(self):
f79be01821c4 Arangodb backend, first version which barely works for reading
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 245
diff changeset
46 # make sure that the results are actually fetched into memory, otherwise we'll pass
f79be01821c4 Arangodb backend, first version which barely works for reading
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 245
diff changeset
47 # a ViewResults instance around which is not what we want
f79be01821c4 Arangodb backend, first version which barely works for reading
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 245
diff changeset
48 return list(Feed.all(self.database))
f79be01821c4 Arangodb backend, first version which barely works for reading
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 245
diff changeset
49
155
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
50 def _retrieveEntriesForSelectedFeed(self, hideReadEntries):
183
ac1de0a467fd clean up
dirk
parents: 182
diff changeset
51 viewResults = FeedEntry.entriesForFeed(self.selectedFeed, self.database)
182
964661153b97 hide read feed entries if configured
dirk
parents: 181
diff changeset
52 if hideReadEntries:
233
e34c53a3e407 fixes from eric's style check
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 218
diff changeset
53 filterFunc = lambda feedEntry: feedEntry.read is False
183
ac1de0a467fd clean up
dirk
parents: 182
diff changeset
54 viewResults = filter(filterFunc, viewResults)
181
1ab48428e2f9 sort the feed entries
dirk
parents: 180
diff changeset
55 return viewResults
155
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
56
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
57 def markSelectedFeedAsRead(self):
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
58 for feedEntry in self.entriesForSelectedFeed():
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
59 feedEntry.markRead(self.database)
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
60
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
61 #
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
62 # handling of the selected feed entry
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
63 #
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
64
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
65 def _markSelectedFeedEntryRead(self):
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
66 self.selectedFeedEntry.markRead(self.database)
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
67
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
68 def toggleSelectedFeedEntryRead(self):
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
69 self.selectedFeedEntry.toggleRead(self.database)
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
70
132
63d6d60d37ff new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
71 def createFeed(self, url):
166
04c3b9796b89 feedparser uses the proxy now if one is configured. To implement this the FeedUpdater had to change a bit - sqlalchemy backend is not yet refactored.
dirk
parents: 164
diff changeset
72 feed = Feed.create(url)
04c3b9796b89 feedparser uses the proxy now if one is configured. To implement this the FeedUpdater had to change a bit - sqlalchemy backend is not yet refactored.
dirk
parents: 164
diff changeset
73 feed.store(self.database)
04c3b9796b89 feedparser uses the proxy now if one is configured. To implement this the FeedUpdater had to change a bit - sqlalchemy backend is not yet refactored.
dirk
parents: 164
diff changeset
74 FeedUpdater(self.database, self.preferences()).update(feed)
132
63d6d60d37ff new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
75
63d6d60d37ff new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
76 def updateFeed(self, feed, changes):
162
ab2b26412b77 setting a feed's properties via the GUI works now
dirk
parents: 161
diff changeset
77 for key in changes.keys():
ab2b26412b77 setting a feed's properties via the GUI works now
dirk
parents: 161
diff changeset
78 feed[key] = changes[key]
ab2b26412b77 setting a feed's properties via the GUI works now
dirk
parents: 161
diff changeset
79 feed.store(self.database)
132
63d6d60d37ff new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
80
164
3eeda7cec39b delete a feed via the GUI works now
dirk
parents: 162
diff changeset
81 def deleteSelectedFeed(self):
240
1b98925facf6 Bugfix: use the existing _retrieveEntriesForSelectedFeed method to retrieve all articles when deleting a feed
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 236
diff changeset
82 viewResults = self._retrieveEntriesForSelectedFeed(False)
164
3eeda7cec39b delete a feed via the GUI works now
dirk
parents: 162
diff changeset
83 for row in viewResults:
3eeda7cec39b delete a feed via the GUI works now
dirk
parents: 162
diff changeset
84 del self.database[row.id]
3eeda7cec39b delete a feed via the GUI works now
dirk
parents: 162
diff changeset
85 del self.database[self.selectedFeed.id]
132
63d6d60d37ff new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
86
146
8ec20377bcb0 move getting the entries for a feed to the backend so that the couchdb backend can use a custom view for feed entries
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 144
diff changeset
87 def entriesForFeed(self, feed, hideReadEntries):
169
91a24f499318 introdue an abstraction for the name of the database so it can be changed via commandline parameter
dirk
parents: 166
diff changeset
88 viewName = CouchDb.feedEntriesByFeed()
146
8ec20377bcb0 move getting the entries for a feed to the backend so that the couchdb backend can use a custom view for feed entries
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 144
diff changeset
89 if hideReadEntries:
169
91a24f499318 introdue an abstraction for the name of the database so it can be changed via commandline parameter
dirk
parents: 166
diff changeset
90 viewName = CouchDb.unreadFeedEntriesByFeed()
148
c5a427d46703 displaying entries for a feed works now with the couchdb backend
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 146
diff changeset
91 return list(FeedEntry.view(self.database, viewName))
146
8ec20377bcb0 move getting the entries for a feed to the backend so that the couchdb backend can use a custom view for feed entries
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 144
diff changeset
92
155
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
93 def markFeedEntriesAsRead(self, indices):
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
94 for index in indices:
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
95 feedEntry = self.entriesForSelectedFeed()[index]
a05719a6175e move common functionality into an abstract backend class, have both backends inherit from it. Implement enough of the couchdb backend that reading feeds (and marking feed entries as read) is possible
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 148
diff changeset
96 feedEntry.markRead(self.database)
132
63d6d60d37ff new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
97
63d6d60d37ff new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
98 def updateAllFeeds(self):
213
524cbf9e413c use correct TODO tags so they show up in the tasks view in Eclipse
dirk
parents: 204
diff changeset
99 # TODO: use a view instead of iterating all feeds
139
2cd30af937fa add the required methods for determining if a feed needs to be updated
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 133
diff changeset
100 allFeeds = Feed.all(self.database)
2cd30af937fa add the required methods for determining if a feed needs to be updated
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 133
diff changeset
101 for feed in allFeeds:
2cd30af937fa add the required methods for determining if a feed needs to be updated
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 133
diff changeset
102 if feed.needsUpdate():
211
1ac0b8e2feae wrap the individual feed update with an exception handler so that a sinlge broken feed doesn't halt the entire update process
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 204
diff changeset
103 try:
1ac0b8e2feae wrap the individual feed update with an exception handler so that a sinlge broken feed doesn't halt the entire update process
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 204
diff changeset
104 FeedUpdater(self.database, self.preferences()).update(feed)
1ac0b8e2feae wrap the individual feed update with an exception handler so that a sinlge broken feed doesn't halt the entire update process
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 204
diff changeset
105 except Exception as ex:
1ac0b8e2feae wrap the individual feed update with an exception handler so that a sinlge broken feed doesn't halt the entire update process
Dirk Olmes <dirk@xanthippe.ping.de>
parents: 204
diff changeset
106 logging.getLogger("FeedUpdate").error("Exception during fetch: " + str(ex))
132
63d6d60d37ff new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
107
63d6d60d37ff new backend for using CouchDB as persistence mechanism ... currently all methods are unimplemented
Dirk Olmes <dirk@xanthippe.ping.de>
parents:
diff changeset
108 def expireFeedEntries(self):
201
50b578588405 method ordering
dirk
parents: 191
diff changeset
109 expireDate = self._calculateExpireDate()
174
d0ced79b5030 implement expiring read feed entries
dirk
parents: 169
diff changeset
110 logger = logging.getLogger("expiry")
d0ced79b5030 implement expiring read feed entries
dirk
parents: 169
diff changeset
111 logger.info("expiring entries older than " + str(expireDate))
d0ced79b5030 implement expiring read feed entries
dirk
parents: 169
diff changeset
112 for entry in FeedEntry.getReadFeedEntriesOlderThan(expireDate, self.database):
d0ced79b5030 implement expiring read feed entries
dirk
parents: 169
diff changeset
113 del self.database[entry.id]