
		 Newsx - Frequently Asked Questions
		 ==================================

Is there a way to make newsx fetch articles more quickly?

	Yes. Try adding the following option:

		--window 10

	You could even try with a larger number.

	This feature should not generate any problems, but is not 
	enabled by default since the method of operation is not explicitly
	mentioned in RFC-977.


Fetching new groups seem to take forever.

	If you are connected to a server with a big news spool, many
	newsgroups may contain thousands of news articles. If you start
	fetching a new newsgroup, newsx will try to fetch all of these.

	It is often convenient to limit the number of articles fetched
	in previously unseen groups, for instance by giving the option:

		--maxnew 100


How do I find out about future newsx versions?

	Send an email message to majordomo@kvaleberg.no with the 
	following message body:

		subscribe newsx-announce

	This should add your name to the newsx announcement list, where
	future versions and possible bug alerts will be announced.


I fetch most of my news from one provider, but in addition I want to pick
up "linux.*" from another source. Is it possible? I can't find any newsx
configuration file where I can specify this.

	That is because newsx relies on the configuration file already
	present in your news spool. This neatly coordinates the actions of 
	the outbound stream prepared by the news server itself with the
	inbound stream handled by newsx.

	Assuming you already has your main newsserver set up (see README),
	you have to define the second news source. For C News, you would
	then add to the "sys" file:

		foo/foo.bar:linux.all/all:FL:

	Similarily for the INN "newsfeeds" file:

		foo/foo.bar:linux.*:Tf,Wfb:

	This will tell newsx to only pick up the linux hierarchy to the
	foo spool, and tell the local news server to place local postings
	for the "linux" hierarchy in the foo spool.

	Assuming acme.net don't carry these groups, you will never get
	any linux news from news.acme.net. But to prevent postings to be
	sent there in vain, and to avoid attempts at fetching these
	groups, the acme entry needs to be modified:

		acme/acme.net:all,!linux.all,!junk,!control/all:FL:

	And for INN:

		acme/acme.net:*,!linux.*,!junk,!control:Tf,Wfb:

	To fetch from both sources, you would need to invoke newsx twice:

		newsx acme news.acme.net &
		newsx foo  news.foo.bar  &


How do I configure a local newsgroups?

	If you define newsgroups that should be local to your own machine,
	you need to exclude these from the external feeds. Assuming these
	groups are called "local.something", the entry in the C News sys
	file should be:

		acme/acme.net:all,!local.all,!junk,!control/all:FL:

	Similarily for the INN newsfeeds file:

		acme/acme.net:*,!local.*,!junk,!control:Tf,Wfb:

	With regards to INN, notice that the trailing ".*" is required.
	The "local.*" could be written as "local*" but this would then 
	also include any "localities" hierarchy, for instance. 


I have tried to set up a local newsgroup, but newsx insists on fetching
the local group from other servers.

	This surely means that there is something wrong with the 
	configuration in your sys or newsfeeds file. Ensure that the
	entry for every external news server excludes the local groups.

	To find out what actually happens, run newsx with a sufficient
	debug level (e.g. -ddddd). You should see something like:

		group local.something disallowed by newsfeeds


Should I use the distribution field for limiting the distribution of
local newsgroups?

	In general, the answer is a definitive no. The distribution 
	field is a mechanism that is really best suited for limiting
	distribution of specific news articles.


How can I automatically ensure that my local spool carries all available
groups, subject to the limitations of my sys/newsfeeds file?

	You can make newsx produce a list of newsgroups that the
	remote newsserver carries, and feed that list to a small
	script that handles the actual task of local subscriptions.

	This can be combined with normal news fetching by adding the
	option: 

		--newlist "|addgroup.all" 

	This will pipe the list of new groups to the script specified, 
	which incidently is available in the lib subdirectory in the
	newsx distribution.

	On servers with many newsgroups this does take some time, so it 
	will be useful not to do it too often. A standalone effort may
	thus be convenient, e.g.:
		
		newsx acme news.acme.net --newlist "|addgroup.all" \
				 --no-fetch --no-post --locks none


How can newsx be set up to handle the increasing problem of spamming?
It surely makes sense not to download the spam article in the first
place.

	The answer to that is not as obvious as it might seem, and
	all depends on how large percentage of the message stream
	that would be stopped by an automated spam filter.

	The streaming mode of operation used by newsx gives a very
	significant transfer speed increase over other methods.

	Filtering means that article transfer must be handled in a
	less efficient manner, either by using the overview mechanism
	(which is not entirely standardized and universally available
	anyway), or by reading the article header and body
	separately.

	This extra overhead means that the spam content must be very
	high to warrant stopping it in newsx.

	There is one exception to this, and that is methods involving
	spam killing based on the message ID alone. The --filterm
	option is provided for this purpose.


Why can't you set up newsx to check for things like multi crossposts on an
article, subject lines like `CASH CASH CASH`, `MAKE MONEY FAST`, `SEX SEX
SEX`, or the the path of an article to exclude known spam ridden sites?

	Doing news spam filtering on an individual basis really is a
	lost cause due to the huge efforts required to keep such
	filters up-to-date with the countermeasures in the spam
	arena. There are also other considerations:

	   1. Lost efficiency due to need for separate article header and
	      body requests.
	   2. Makes streaming much more difficult and less efficient to
	      implement.
	   3. Not as much gain as expected anyway, since article headers 
	      easily make up a significant part of most articles anyway.

	So, other methods of spam filtering is really more suitable:

	   A. Use an ISP that does spam filtering.
	   B. Use NoCeM to stop spam.
	   C. Do spam filtering in the local news spool, using existing
	      methods.
	   D. Do spam filtering in the newsreader by use of kill-files and
	      article scoring mechanisms.


How can newsx be set up to take advantage of the NoCeM effort?

	The c-nocem package from Olaf Titz can be used with both INN
	and C News. It is available via:

		http://www.inka.de/~bigred/sw/c-nocem.txt

	c-nocem will delete articles that already has reached the
	spool. Just as importantly, it will mark articles not arrived
	yet in the history database.

	Since newsx refers to the history database for every article,
	this does in fact mean that newsx will NOT fetch any articles
	that has already been flagged by NoCeM.

	This method will work better the fresher NoCeM info you can get.
	It might be sensible to fetch and handle the NoCeM newsgroups
	before starting a new ordinary transfer.

	Note that the volume of the NoCeM newsgroups is pretty huge.
	For a small spool, this will mean that the total news
	transfer volume will most probably INCREASE when using NoCeM.
	NoCeM may still serve a useful purpose in reducing the number
	of undesired articles popping up in the newsreader.


I want to limit the number of groups that I actually fetch from a 
particular server. Should I use the -e mechanism for that?

	It depends. It will usually be better to set up your requirements
	in the sys/newsfeeds file. For instance by splitting into several
	incoming spools for a single news source.

	But by all means, the -e approach does work. You must manually
	edit and add a tag line in the server active file found in 
	"/var/spool/news/in.hosts/spool", where "spool" is the name of
	the spool that appears in the sys/newsfeeds file. Rearrange it
	so that all groups you want appears before the tag, and all those
	that you don't want appears after it. The tag line could look like
	this:

		:end

	Remember to use the corresponding command line option:

		-e end

	The thing with the -e option is that since you have limited the 
	number of groups, newsx won't be allowed to start fetching new
	groups that appear in the active file. So you can't simply add 
	groups with addgroup, you will also have to manually create an 
	entry for them in the in.hosts file.


Why did the configure script fail to find the news home directory?

	The configure script goes through a list of probable directory 
	names in some order of priority to find the one you are using, 
	looking for the either an "bin/config" (C News) or 
	"innshellvars" (INN) file.

	One reason can be that the directory name you are using is not
	on that list. See the variable VISIT in configure.in

	If your system is using another location, run configure with
	the option --with-newshome="/my/news/home". Or set the environment
	variable NEWSHOME before you invoke configure. Please alse send 
	the relevant information to egil@kvaleberg.no, so that future 
	versions of the newsx configure can be updated accordingly.

	Another reason can be that the configuration file is missing
	altogether, which must be corrected because it will cause trouble
	with other news software.

	It could also be that access to the news home directory has been
	prohibited.


Why did the configure script come up with another news home directory than
the one I am using?

	Presumably, the script have found some traces of a previous news
	installation, or perhaps some symbolic link to the actual news
	directory. Try specifying the news home explicitly with the
	--with-newshome configure option. In the latter case, you may get
	away with keeping the configuration as it is.


Why did the configure say that the configure file is "inaccessible"?

	Usually because it *is* inaccessible. It is commonplace to protect
	access to this configure file from other users than he actual news
	user. The solution here is to install the software as user news, 
	or extend the read permission of the configuration file.


Why did the configure script say I am using INN when I am actually using
C News, or vice versa?

	Presumably, the script have found some traces of a previous news 
	installation, and has become confused.


Why did the configure script come up with another news spool directory 
than the one my system is in fact using?

	The news spool name is read from the news config file. In C News,
	the file is "bin/config" and the variable is NEWSARTS, In INN, the
	file is "innshellvars" and the variable is SPOOL. Either way, if
	this doesn't match the actual directory you are using, there is
	something wrong with the setup of your news server software. 


Why did the configure script come up with another history file than the
one my system is in fact using?

	The reason is most probably that the history database is in a non-
	standard location. 
	
	If you are running INN, this should have been defined properly by
	the news configuration file, so there must be something wrong 
	with news server setup.
	
	If you are using C News, you can either supply the configure
	script with the --with-history=FILE option, append a (non-standard)
	definition of HISTORY to "bin/config", or solve this by
	installing appropriate symbolic links for the files "history", 
	"history.dir" and "history.pag" from the news home directory.


Newsx compiled fine, but doesn't work?

	Run newsx from the command line, adding the option "-dddddd".
	You will then be told what newsx does step by step, and
	finding out what happens should be self-explanatory ;-).


How do I install the binary and manual together with the news binaries?

	By default, newsx is installed in /usr/local/bin and /usr/local/man,
	respectively. To install it in NEWS/bin and NEWS/man, try these
	configure options:

		--exec-prefix=NEWS --prefix=NEWS


How do I make the newsx binary as small as possible?

	Instead of "make install", try:

		make install-strip


Newsx seems to fetch news without complaint, but when I start up my
newsreader I don't see anything?

	The articles are probably still residing in the 
	"/var/spool/news/in.coming". To move these files to the news
	spool proper, you must do in C News:

		newsrun

	And in INN:

		rnews -U


But the incoming spool is also empty?

	The local server has probably rejected the articles, in which 
	case you will find them in "/var/spool/news/in.coming/bad". 
	Have a look in the logs to find out why they were rejected.
	For C News, look at "/usr/lib/news/errlog".

	When you have corrected the configuration, do:

		cd /var/spool/news/in.coming
		mv bad/* .

	Finally rerun newsrun/rnews as shown in the previous question
	to try again.


How can I verify that history database lookups are indeed working?

	Find the message ID of any article in the newsspool, and feed it
	to newsx via the -q option. The response should be "yes". Likewise,
	any non-existing message ID should respond with "no".


Why doesn't the history lookup function work?  

	The reason is probably that the history database is in a non-
	standard location, see the question about history configuration.
	
	It could also be that the database format is wrong. The standard
	configuration assumes that the news server uses the "dbz V3.4" 
	database manager, or the "dbz V6.1.1". If your system has been
	configured otherwise, you have to manipulate the configuration 
	variables DBZ_VERSION, HAVE_LIBDBZ and HAVE_LIBDBM.


What shall I do? The "ps" process status doesn't work, and/or crashes
the system.

	Give the --without-ps-status option when running configure, and
	re-make. Also, please run config.guess and report what it says to
	egil@kvaleberg.no so that the configure script can be made to
	disable this feature by default for future releases for the system
	you are using.

	The dynamic process status probably has some portability   
	problems some systems, but is simply too useful to take away.


What on earth is this tiny.server this in included in the distribution?

	One of the special features of newsx is that it can fetch
	news using NNTP relayed over Telnet.

	The point of the tiny.server is to offer people who can log
	onto machines that does not offer any NNTP connections a way 
	of fetching news. Since tiny.server is a Unix shell script, you 
	don't even need access to a C compiler. 

	Note that the shell script idea makes tiny.server quite         
	inefficient. So tiny.server mostly exists for its hack.value.
	Also, only a subset of NNTP is offered.


Does newsx figure out who to send outgoing news to in the same way that
it downloaded news?

	By design of the news system, it is in fact INN (or C News) that
	decides on the outgoing news. newsx simply sends on whatever appears
	in the outgoing queue for the external news server. 

	On the other hand, it is newsx that decides what to fetch. The reason
	why this appears to be a symmetric process is that newsx interprets
	the newsfeeds (or sys) file in the same manner as the rest of the
	news system.


I got a "ctlinnd flush failed" message. What does that mean?

	When running with INN, newsx performs an INN "ctlinnd flush"
	operation. If this operation fails, the above message will appear.
	Any error message from ctlinnd should appear just before this
	message, e.g. "No such site". Consult the INN documentation for
	more information: "man ctlinnd" might be a good start.


 In the INN outgoing spool directory, I have found a file with a ".togo"
 suffix. What is that for?

	The file with the ".togo" suffix is used by newsx to handle retries
	of article posting. The exact procedure is:

	 1. Apply lock for other newsxes
	 2. Open outgoing spool
	 3. If a spool.togo exists: rename/append it to spool.retry
	 4. Rename the spool file to spool.togo
	 5. Do a ctlinnd flush spoolname
	 6. If a spool.retry exists: Append it to spool.togo and erase
	 7. When posting: Any articles with problems go to spool.retry
	 8. When finished: Get rid of spool.togo
	 9. Rename spool.retry to spool.togo if it exists


I am trying to understand how newsx is compiled, but I find the Makefile and
configure script very hard to understand. Why?

	That is because these files were never meant for for human 
	consumption or maintenance. They were produced by the GNU tools
	automake and autoconf, respectively. The real source files are
	Makefile.am and configure.in, which should be a bit more readable.
	Especially if referring to the automake and autoconf documentation.
