OT: User forum design

[D Yuniskis]

Hi Joseph,

JosephKK wrote:

On Wed, 07 Apr 2010 18:39:48 -0700, D Yuniskis <not.going.to.be@seen.com> wrote:

I need to design a "User's Forum" for an upcoming product release.
The goal (as always) is to let users share their ideas, solutions,
gripes, etc. with each other directly without going "through"
any business structure.

Interesting task.

<frown> Annoying task. But, the product is really interesting
so its worth the effort to get it right. :-/

There are two well done webforum & NNTP
combination systems that i have seen: OpenSuse forums and
Microsoft public forums. Both are actively managed. Yahoo

By "managed", I assume you mean there is a human overseeing
their operation? (moderator, of sorts) Sorting out the
spammers, abusers, etc.?

forums also appear to have SMTP gateways, but they overdid
the registration thing. Registration is a good idea anyway,
it could require reading an email and using a key (good for

This is a closed list. Only folks owning the product
get on it. It isn't visible -- or accessible -- otherwise.
Granted, an "owner" could have a bot accessing it on his
behalf. But, if that bot misbehaved (however the rest of
the subscribership defines "misbehaving"), then it's
out!

a year?) in order to post (anti-bot). Regular spam blocking
will also have to be considered, as well as automatic

Spam from "foreign" sources is gated by membership list.
If a subscriber starts posting spam, then other subscribers
boot him/her from the list. (does buying another instance
of the product give you another subscription??)

blocking of profanity. Take a look at STUMP robomoderation
program and group distributed moderation. For people that
wish to continue to private email they should have the wit to
take anti-harvesting precautions (don't block it, but don't

I think you just scrub the email addresses (and "real names")
from all posts. Its trivial to map each real user into
an arbitrary (though consistent) "anonymized user".
I.e., the only thing you see is the message content.
If you are observant, you might notice a trend in some
of the questions from a particular "anonymized user".
Some information could leak that way.

Of course, there is nothing to stop folks from posting
(in the message body) their real name, email address, etc.
(You can't prevent people from shooting themselves in
the foot! :&gt

encourage it). Provide a FAQ and post some policy basics
regularly. The FAQ should contain some "how to make a good
post" information.

 

[D Yuniskis]

Hi Joseph,

JosephKK wrote:

On Thu, 08 Apr 2010 00:46:39 -0700, D Yuniskis <not.going.to.be@seen.com> wrote:

Hi David,

David L. Jones wrote:
D Yuniskis wrote:
You missed the criteria. *No* moderator. I.e., I am
looking for a scheme whereby the users can moderate the
forum (hence my reference to craig's list). So, any
members misbehaving simply become un-members (i.e.,
strong incentive to behave as product updates are
distributed via the same channel).
Sorry, I've never used Craigs List, but how can a user get booted without
someone deciding they should be booted?
Craigs List allows readers -- WHO CAN BE ANYONE -- to mark
posts as inappropriate (for a variety of reasons). Some
number of independant markings cause the post to be removed.
(I've never explored the actual criteria they use).

SNIP
OTOH, they have to "participate" even if they only want
*read* access ("membership" is given to customers only).
Trivial to do with BBS forums, you can make them membership only.
Sure, old technology. Modern BBS's are an outgrowth of much
that originated in mailing lists.

Then perhaps what you want is an old (over 20 year ago) BBS that
pretty much does what you want, and slap a web or SMTP interface
on it with LAMP. Email translators are old hat as well.

Software is easy. The tough part is figuring out how it
should behave. What it should look like to the user so
that it is easiest to use. Remember, this isn't the sort of
thing most folks are going to access frequently. And, when
they do, they are going to *need* "an answer" -- they won't
be wanting to play around trying to figure out how things
work...

You could also split between which parts of the system are pull and
which parts are push. E.g. conversations are pull, notice of update
is push.

Hmmm... I'd have to think about that. There are some tricks
I might be able to play.

 

[D Yuniskis]

Hi David,

David L. Jones wrote:

D Yuniskis wrote:
Hi David,

David L. Jones wrote:
Yes, but you usually don't get that with BBS forums. Every one I
belong to (and run) is civil and on-topic. No spam, no bad behaviour.
Shirley, you jest?

Not at all, check them out for yourself:
www.eevblog.com/forum
and
www.calcwatch.com/forum

Neither has any bad behaviour or spam.

These aren't "product support" forums (or, do you consider yourself
the "product"?).

On a blog, people expect pontification, expostulation, etc.

On a product support forum, folks expect answers to questions
concerning the use, design, repair, enhancement, etc. of
the *product*. They don't want to see posts about politics,
religion, Gucci handbags, Toyota recalls, etc.

It's not a "social site" -- anymore than you call Tech
Support to chat about the weather, etc. (imagine your
reaction if the support guy wanted to talk about the
NBA playoffs -- even if you *weren't* being charged
by the minute! "Hey, buddy, that's great. I hope your
team wins. But, I really want to get this gizmo working
so I can get on with my life...")

And, people don;t tend to "linger" (except the few who want to
be "gurus"). It's more hit and run -- ask a question,
(hopefully) get an answer, *possibly* hang around long enough
to say "Thank You"... then, disappear until the next time
you find yourself scratching your head.

They are just two that I run, but it's the same thing on almost every other
forum that I frequent.
People are always civil enough to post in the appropriate group category.

I've been wandering around looking at
various fora and see quite a lot of OT posts. Folks who
just seem to "want someone to talk to". Would you call
Tech Support "just to chat"? Would you call your plumber
to chat about a problem you are having with your car?

Oh, for goodness sake, people like to talk on forums!
Sometimes it gets off topic etc, what's the big deal?, that's the nature of
the beast.

*I* don't really care. But, subscribers *do* -- to varying
degrees. If subscribers feel a forum is wandering off topic
too much or is filling with "silly questions", they want to
be able to act on that. They don't want to be the tail
being wagged by someone else's dog -- it's *their* forum.
People abandon resources as their information content
(and perceived value) diminishes.

If folks *want* (or are willing to tolerate) things going off
topic, then they -- by their actions (or inactions) -- will
set the policy for the forum (i.e., by not "censoring" posts
or subscribers). If a self-appointed "BofH" gets too
aggressive in trying to "set policy", he/she may find themself
"on the outside looking in" :>

That;s the whole point of empowering the subscribership to
maintain the forum instead of some "owner".

E.g., I am very careful, here, when I discuss problems
to make sure I don't disclose anything that would betray
a client's confidence (or get me fried for violating an
NDA!). Since I can't know or control who reads my
comments, I often have to "speak in parables", etc.

(i.e., I don't think Toshiba would want folks reading
about "gas pedal questions" ~2 years ago... if you see
the parallel)

Almost every web host has these available as single click installs
these days, otherwise it's pretty easy to donwload and install
yourself. Trivial to set up and most users are familiar and
comfortable with
forums like these.
You're presupposing you understand my application and
user base better than I. :> Give me credit for having
evaluated all the *obvious* solutions -- hence my post,
here. If I'm looking for a five-wheeled vehicle, *assume*
I *know* that I need a five-wheeled vehicle...
Ok. But it is still completely unclear why you think a BBS forum
will not work.
BBS forums are pretty much the standard. People know them and
understand how they work. You'd want a pretty good reason to use
something else for a user type forum, and you haven't really
explained why a BBS is not suitable.
A BBS (assuming we aren't talking about accessing that via an SMTP
gateway) is a pull technology. This means it has to accept
requests whenever the user wants to access it. As such, it's
hosting must be 24/7/365 (or, some other established set of
"business hours").

Yeah, that's how countless forums operate. What's the problem with that?
If it goes down for a few minutes or an hour every now and again, no one
really cares.
I'm not talking about SMTP access, I'm talking about a BBS forum people
visit.

It means the service has to be "exposed" 24/7. Even if no one
is using it. It means it is vulnerable 24/7 to exploit. It
has to wait for folks to come to it -- instead of doing its
work and going *to* them.

Any server that has to be available "at will" is also exposed
to hackery as it *is* accessible. Consider how much easier
it is to protect an SMTP interface than an entire *host*!
Traffic to the "server" servicing the "group" can be
controlled by the server itself (i.e., *it* can decide when to
go fetch its incoming mail and when it wants to *send*
its outgoing mail). Imagine how much harder it is
to hack a "Mail Station" than a "general purpose PC"...
(the former has a very narrowly defined interface whereas
the latter has to be adeptly managed to ensure it has
the smallest visible interface possible)

And, it never knows how *many* users will want to access it at
any particular time. As load increases, performance degrades.
This degradation is directly visible to the user -- the
interface becomes sluggish. (use web mail and see how
performance varies throughout the course of the day!)

Really?
I run half a dozen high bandwidth sites with databases, streaming video, and
two forums on a simple <$10/month shared server and there is no performance
degredation on essentially any aspect of it, let alone the forums.
Downtime is almost zero.

And you are a commercial site? Or, just someplace that
folks drop in on? Anything "of value" on those sites?
Or, just fluff and advertising? Anything AT STAKE??

[snip]

Gawd, I'm not even going to read all that you wrote there, it's getting too
much.

*You* questioned the criteria I laid out for the problem.
I have tried to allude to some of the types of issues that
drove my establishment of those criteria. Note that I
explicitly mentioned:
"A few issues are not negotiable; some have 'very high
inertia' (but could be 'moved' given the right impetus)
and others are just 'whims'."

Of course there is a lot there. It's not an arbitrary
decision: "Let's use some OTS software and set up
yet-another-forum". There are ramifications to the
decision that affect the product, its design, support
and its user base.

There's about 5 man-months of effort (discussions, specifications,
interviews, analysis) predating my post, here. You want that
summarized into a single *short* posting? :> And, upset because
I've already ruled out your "obvious" solution?

But I really get the impression you are MASSIVELY over analysing this whole
thing, really.

Good luck getting your ideal solution. If it was me I'd just stick up a
private BBS forum and be done with it.

I stated CLEARLY in my original post that I was looking for an
SMTP solution. As is so often the case, instead of trying
to address the problem in the solution space I proposed, you
opted to question those criteria.

Remember, I know much more about the product and its users
than you do. :>

Assume I am competent. (my clients do :> )

Assume I approach each decision with an engineering mindset
with criteria that have been set forth to me by those clients.
That I have already evaluated other options and found them
to be inappropriate -- for whatever reason (instead of asking
me to justify all of my criteria)

Assume that if I have set forth some criteria for a solution
here, that those are legitimate and based in fact.

If your employer tells you he wants a device that does "X",
do you require him to justify "X" instead of "Y"? Do you
analyze the market research and the conclusions made from
that research? Do you insist on interviewing prospective
customers?

[N.B. I actually *have* done all of those things in this
case -- but, that's what I'm paid to do :-/ ]

Or, do you try to come up with a solution that fits the
problem space as outlined?

I'm trying to come up with a solution that fits the
criteria I have laid out. That offers the features
that I need. If there was an existing solution, I
would have already adopted it. As this thread
suggests, its not a simple problem to solve. Blame
my client! :<

 

[D Yuniskis]

Hi Paul,

Paul Carpenter wrote:

SMTP driven needs quite a bit of processing on each message
to remove all sorts of problems.

Sanitizing mail is no harder than sanitizing posts on a BBS.
You only have to sanitize each incoming message *once*. I
suspect there are probably more risks to be protected against
in adding "arbitrary data" (text) to a web page than to
a mail message (? this may be an even score as many mail
clients effectively understand and respond to "stuff" in
the body of the message :< )

A BBS (assuming we aren't talking about accessing that via an SMTP
gateway) is a pull technology. This means it has to accept
requests whenever the user wants to access it. As such, it's
hosting must be 24/7/365 (or, some other established set of
"business hours").

Which considering the costs of webhosting these days is cheap and
insignificant to set up, as even your own server with whatever on.

And we NEVER hear of hosts on the 'net being "compromised",
right? Never hear of big firms having data stolen off
their servers (gee, don't they *know* there are risks??)?
Would *your* employer (or clients) let you set up a web site in
the company's domain? How much grief would the IT department
give you?? How much overhead (administrative) would
*inevitably* result?

Any server that has to be available "at will" is also exposed
to hackery as it *is* accessible. Consider how much easier
it is to protect an SMTP interface than an entire *host*!

You are joking aren't you?

An SMTP interface STILL means you have to close off other
services and ensure that even DOS attacks, by flooding a
closed http, telnet or other port does not slow service.
It still has to receive packets to reject closed ports
which takes up bandwidth restricting your valid service.

I think you misunderstood my approach. There is no "exposed
server". Just a box that emulates what a human being could
otherwise do. I.e., open mail addressed directly to it,
copy body of message into a new/empty message, mail that
new message to a group of anonymized subscribers. When
all messages processed, bring the *interface* down -- you
aren't talking to anyone so why leave it "up"?

It's not a service in the classic sense of the word. It
is, instead, just an agent. It looks like a POP3 client
*and* an SMTP client. The service that it provides is
implemented within those confines.

By contrast, unless you get a dedicated box co-lo'ed
just for you *stripped* of all "nonessential services",
chances are, that webhosting service you outsource to
has all sorts of cruft running on their server. They
*want* to have lots to offer. And, each of those offerings
is more code that can be buggy, exploited, hacked, etc.
Some other "virtual host" running on that server can
be compromised and now you are at risk. OTOH, something
would have to travel up the pipe through the POP connection
(which is initiated when *my* "agent" choses) in order to
put my agent at risk. The gate is much narrower and
more well defined.

DoS attacks are a fact of life for *anything*. Having
a fixed presence on the 'net just makes you a *sitting*
target. OTOH, if you are just a *client* that connects
to the network unpredictably, someone has to know *where*
you are and *when* you are there. You can *move* whereas
a server sitting at a fixed symbolic address (DNS/DDNS)
or static IP can always be "found".

You could only get better protection by using a SMTP
service on dial up to do a pull, to reduce visibility.
Then again a ping attack with 64k blocks of data will slow
you down.

You don't have to resort to dialup. That just throttles
the connection. Note that my "agent" can ignore *all*
traffic until/unless it *wants* to talk. It needn't
sit on any connection forever. And, doesn't need
to have a dedicated IP. It can look just like a mail
client -- that fetches mail directed *to* it and sends
out mail *from* it.

Traffic to the "server" servicing the "group" can be
controlled by the server itself (i.e., *it* can decide when to
go fetch its incoming mail and when it wants to *send*
its outgoing mail).

Only if it is on Dial up, SMTP sits there waiting for messages
just like HTTP servers sit there waiting for web requests.

You only need SMTP on the box *to* which the agent connects
(for outbound mail delivery). The "forum" itself need not
reside/execute on that machine. Just it's "published address"
(i.e., the email address to which users direct their posts).

I think you do not understand differences between sending email
from a client using SMTP and the receiving end using SMTP, it
has limited choice on receiving email. You would need somebody
else's server to receive it and then fetch it from there like
POP/IMAP works for email CLIENTS.

Exactly!

The "forum agent" connects to an (inbound) mail server using POP
(or IMAP) to "fetch all mail addressed to FORUM@SOMEDOMAIN". It
"reads" those messages (from folks with real email addresses).
It rewrites the headers (to obfuscate the actual senders'
email addresses/names) and then distributes the rewritten
messages to the forum's subscribers -- by connecting to its
(outgoing) SMTP host (again, as a CLIENT) and pushing the
messages out. This is just a classic mailing list with the
addition of originator obfuscation. The other features that
I mentioned -- you know, my ORIGINAL QUESTION -- are the
real gravy!

The forum agent need not store messages (unless you wanted to
maintain an archive) or any state regarding who has read
each message. That is offloaded to the users' mail clients.
All it has to do is rewrite messages and send them back out.
It exploits the store-and-forward technology inherent in
mail handling (which differs from the connection oriented
nature of HTTP sessions).

Do you realise how many scripts you are going to have to run to
sanitise the emails, to remove unwanted types of headers like
Reply Receipt and read receipts, let alone attachment, character
set, virus/malware scanning and other isues.

You *don't* have to do that on a BBS? C'mon, who are you
kidding? Content is content. If you have to sanitize it in
mail, then you have to sanitize it to at least the same
extent in a web based presentation! If your HTTP based
forum allows attachments, then they have to be sanitized
just the same. If you disallow attachments there, then,
to be fair, you can choose to disallow them in a mail-based
implementation.

The biggest issue is handling *deliberate* offensive or
inappropriate content -- which is why I brought up the
self-policing aspect. Controlling that algorithmically
seems to be an unsolvable problem :-/

What format are the emails to be as lots of emails get sent
as plain text and/or HTML/RTF or even ascii encoded binary only.

Having of old run mailing list for technical users who could not
drive their email software, and remote email servers around the
world misconfigured, the work involved was not trivial.

I've run several lists in the past which was what inspired
this solution. People have email, now. If it doesn't
work, it's something *they* fix -- because they depend on
it for other communications. If your HTTP based solution
breaks (virus, bug, etc.) then *you* have to fix it -- the
users are at your mercy.

How many web sites are "requires browser XXXX version YYY"?
Or, "best viewed at XXXX rsolution"? Or, unfriendly to
users with disabilities? I.e., all browsers are not created
equally.

A browser has to do more than a mail reader. It
stands to reason that there will tend to be more problems
(bugs, inconsistencies) with browsers than mail readers.
The simpler you can make the "forum service", the more
reliability you can expect from it.

Imagine how much harder it is
to hack a "Mail Station" than a "general purpose PC"...

Sorry the vast majority of 'mail stations' are general purpose

No, I meant "mail station" as in "a network appliance
designed solely to send and receive mail". I.e., it
*just* talks POP/SMTP as a client of POP/SMTP servers.
It doesn't/can't do anything else (i.e., that's what the
"forum agent" can look like from a gazinta/cumzouta point
of view.)

Here's the first hit google came up with:

http://the-gadgeteer.com/1999/11/22/mailstation_review/

(No, I am not proposing designing/building such a device.
Rather, I am noting that it's interfaces are exactly those
that the forum agent would have/use -- nothing more... not
even the screen and keyboard!)

PCs, maybe in a server hardware configuration, running some form
of Nix, Windows or Windows Server, using the standard network
stacks to talk to email server software like sendmail, postfix
qmail, ftgate.........

See above. If *all* you have to do is talk to *a* host
(or pair of hosts) provided by ISP/employer then you can
develop a very narrowly defined system that doesn't carry all
that extra baggage/liability. E.g., you don't need a
generic stack, you don't need to support lots of
connections, you don't need lots of horsepower, etc.
Your role is *just* to pull in messages, massage them
and spit them back out.

No, you can't play Freecell :> Or, Empire!

So they have the same and MORE issues of security.
I think you are confusing SMTP as you see for sending email and
how SMTP has to work on a server, i.e. be normally available
24/7/365.

No, you have assumed that the "forum agent" has to reside
*on/in* a "mail server" and *act* as a classic "server".

Look at my FORUM DETAILS post, elsewhere in this topic.
Hopefully, that better explains the model I am proposing.

(the former has a very narrowly defined interface whereas

Standard network interface be it ethernet or dial up via PPP, that
ends up as TCP/IP to the email server software.

You can generally ping mail stations.

the latter has to be adeptly managed to ensure it has
the smallest visible interface possible)

That is true of both 'types' of system as they are basically the
same.

Again, you've missed the model I am proposing.
If you think they are the same, consider how you would
implement my model using HTTP as the transport protocol.

And, it never knows how *many* users will want to access it at
any particular time. As load increases, performance degrades.
This degradation is directly visible to the user -- the
interface becomes sluggish. (use web mail and see how
performance varies throughout the course of the day!)

Delays on SMTP throughput are possible when you get to thousands

Sure, but you can stagger how you handle those users.
E.g., you can spend *hours* sending mail. OTOH, if you
are web based, the user isn't going to wait hours for
the HTTP request to be satisfied.

of users and making sure you are not seen as a Spamhost is sometimes
difficult.

OTOH, using a "push" technology (like mail), allows the
service to spread its workload out over time -- scheduling
outbound mail and processing incoming mail as it sees fit.

Response time to seeing messages posted appear, makes the
service appear sluggish.

And an overtaxed web host looks just as sluggish.
People don't *expect* instantaneous responses when
it comes to mail. Even email! :>

OTOH, people get pissed if they have to wait 15 seconds
for an HTTP connection! Or, "refresh" the page after
5 seconds because they think something is broke
(packet got lost, etc.). Think about it. Think about
how *you* use each of these services and what your
expectations are of each.

Mail *can* be delivered "almost instantly". Yet, do
you send a message (to a friend at work) and then
sit there waiting for a reply to it? If it doesn't
appear in 30 seconds, do you resend it?? Or, do you
move on to some other activity and "check back later"?

OTOH, when you go to a web site, you *expect* it
to be "nearly instantaneous". You grumble if the
web page loads slowly, etc.

Or, am I the *only* one who behaves this way?? :>

And, if you are posting a product support question,
all you are going to see is your post -- until someone
else stumbles onto the (HTTP hosted) site and opts
to answer it. So, you *still* have to wait for a
reply -- even though the site is "online" and always
accessible. (you can "check your mail" for a reply
at least as easily as you can go to a web site -- even
if it is bookmarked -- and "login"; then browse to find
your post and see if there are any followups!)

And, it can leverage a multitude of other hosts -- the
mail hosts for each of its subscribers -- to act on its
behalf (by contrast, a pull technology has to carry all
of the load by itself).

You really do not know what you are talking about.

Defend your accusation. I have claimed that my "forum
agent" (running off a PDA in Starbucks coffee shop)
can send *a* message and that it can then rely on
dozens or hundreds of other machines around the planet
to store and forward that message, as needed, to
reach each of the individual addressees in that message.
I.e., they "act on its behalf". And, as such, "leverage
a multitude of other hosts".

Even when the messages have reached their ultimate
mail hosts (for each of the subscribers), those hosts
are *still* acting on my behalf -- by buffering the
message(s) until the users connects to their mail
hosts and retrieve their mail. My "PDA forum agent"
could be powered off and sitting in my glovebox by
that time...

[I use the PDA as an example as it shows how limited
the role that "forum agent" can take! I am not
advocating this method of implementation -- though
there is nothing that precludes it!]

Consider that the "server" can send one copy of a message
to a multitude of recipients in a given MX. The "remote"
MX can then handle distributing those copies to the N
recipients that it services (instead of waiting for those
N recipients to come *to* the "server").

You are talking about using somebody else potentially as
a recognisable Spamhost. How to get your ISP to cut off your
service.

TI sends out how many thousands of email messages each day
telling you about there latest and greatest chips that they
are promoting "this week" (tomorrow, there will be something
equally grand and wonderful -- at least in THEIR opinion).
Are *they* a spamhost? Do you think the folks who
provide network access to TI are going to cut off their
service??

Consider how heavily MS's servers get pounded whenever
a new service pack is released -- everyone wants the
update *now*.

Only those who blindly update, "if it aint broke don't fix it"

Real serious users actually wait for the masses to do MS's
using the customer to test the release first.

You're missing the issue. With a "pull" service, you
are at the mercy of your users. You have to ensure
that your service is sized to meet whatever sort of
"peak" demand you might likely encounter.

If, for example, you announce an update to the device's
firmware that fixes a critical bug or adds some desirable
functionality, do you think folks are going to say,
"Gee, I'll wait a few weeks until the mad rush is over"?

Instead, they will try to fetch the update. If they
are met with a "Sorry, our servers are temporarily
overloaded. Try again later" message, they will grumble.
If they get lousy download performance (also because
the server is overloaded or the pipe is full), they
will grumble.

If, instead, there is a message sitting in their mailbox
with the new firmware image there for them -- or, if there
is a message telling them "Reply to this messageif you
would like us to SEND you the new firmware image" -- they
will either install it *now*, or leave it sit there
KNOWING IT ISN'T GOING AWAY util they have the time to
install it.

In either case, *you* schedule how and when you will
deliver that to the user instead of having to be ready
to give it to him/her the *instant* he requests it.

Again, the mail paradigm works in your favor -- the user
*expects* some (short) delay in receiving mail. As long
as messages don't get *lost*, he will typically wait
patiently. No need to keep trying the web site to
see if things have improved...

OTOH, if the updates were *pushed* to
users, then MS could better manage the problem. Any
effort "waiting" on a server is waste -- especially
if you have to come back later to get what you want
(and still have no assurances of being able to do so!)

The overhead of doing that by things like SMTP is ridiculous
as the method does not support binary transmission, you have
to ASCII encode your binary, bloating it and taking longer
to send.

Ah, yes. And HTML is *so* terribly efficient! C'mon,
think about what you're saying... Everything is ASCII/UTF8.
Plus, damn near everything has an HTML tag associated with it.
And, possibly some scripting embedded within (Gee, my browser
doesn't support this version of Java...). And, of course,
all the decorations that accompany the page (isn't google's
front page refreshingly sparse??! Of course, there's nothing
*there* of interest besides the image-of-the-day :-/ ).
Plus, all the links to other pages "just to be helpful".

And, lets not forget the inevitable ads that alwasy seem to
creep in -- even if they are for your company's products!

Or, are we now saying user must support HTTP for the
BBS access and FTP to pull down files? Any other
protocols you want to add?

A BBS requires state be kept for each subscriber -- i.e.,
which messages he/she has read, which ones he/she has
replied to, which threads are of interest, etc. A mail
client handles all of that overhead "magically".

For weird ideas of 'magically'.

How so? To the user, all of this stuff *is* handled magically.
They don't have to explicitly mark each article as read. Or,
which they have replied to.

BBS does not have to do all of those things if it does not
want to,

Sure, so you're changing the design requirements of an HTTP
implementation vs. mail because the feature isn't easy to
implement?

on a web interface it could just use the visited link
colour changes to do that.

Yes, you can track which links are visited and not
visited. But, then you can never reset your history
lest you lose *all* of this information! I.e., if you
want to delete all your *other* history -- yet preserve
the links which have been visited ON THIS SITE, then
you have to manually go through and clear all those
sites while taking care not to clear *this* site.

Conversely, if you want to clear this site -- OR SOME
PORTION OF IT ("Gee, this message is worth rereading!
Let me unmark it so I see it again as "unread" next
time I come here!") -- then you have to go through
and manually clear just those portions of the history.

And how many users do you expect to know how to do this?
Contrast that with email users...

You also can't "delete" (i.e., "I don't need to EVER
see this again") a link (message). You just have to
remember not to look at it anymore. In an email client,
you just delete the message and its gone.

You can't flag something as "interesting".

You can't converse with *a* user (i.e., send a reply
to a particular question that you might not want to
share with others that you know are watching)

*If* you wanted to share some message with another
party (non-subscriber), you have to cut and paste
from the web page. You *might* be able to "email
this link to a friend" -- but, then you can't annotate
it contextually.

The BBS paradigm doesn't lend itself to thinking
in terms of individual correspondence. Instead, they
are groups of threads. You can't see where you left
off and what's next -- not anywhere as easily as you can
with mail. BBS is meant for people to "graze on".

Mail can be read via more interfaces than web pages
can be browsed. E.g., you can read/send mail from a web
based interface; you can't read web pages from a mail
client! I've tried viewing several BBS's from a PDA
and a smartphone and found both lacking (mail is "bad"
on these devices but BBS's are abysmal as they have to
fit within the context of a "web page"). I see this
platform as a definite trend.

People moved away from discussion mailing lists as the general
method of discussion about 6 years ago, as everybody went to web.

People also moved away from beginning correspondence with
a "salutation". Is that an improvement? Email and News
used to be strictly ASCII. We now have HTML. Improvement?

Lots of HTML and other emails are not displayable easily on PDAs
either.

You can strip HTML from mail. Kinda hard to strip all the
HTML from your web based BBS and view it as "just text".

Providing "one to one" conversations via a BBS is clumsy.
It's "something extra" that has to be implemented
(maintained). OTOH, doing so with a push technology
is trivial -- process the incoming message, rewrite the
headers and pass it along to the intended recipient.
There is no need to store it (state) on the BBS/server.

A mail client has an intuitive interface through which

In my experience there are a lot of users who cannot effectively
use their email clients, and you get 95% of users 'top posting'
without trimming so as a thread progresses your bandwidth
requirements increase exponentially.

And that NEVER happens in a BBS? I guess these very same
people magically learn how to behave when faced with a
web based interface hosted from afar -- vs a (often
graphical) mail interface hosted locally? People don't
trim posts (or email) of extra quoted stuff. Some people
even *smoke* and cross *between* intersections.

A "growing message" costs just as much bandwidth
whether "I come to it or it comes to me". Bytes is
bytes. And, all those HTML tags just make the
BBS posts that much more bloated as they come down
the wire.

If the forum's subscribers decide that a certain behavior is
unacceptable, they can coerce compliance -- by eventually
booting the offender off the forum. I can resubscribe to
a BBS as many times as I get booted off :-/

a user can preserve or ignore topics of interest (as
well as tracking threads that he/she is participating in).
Messages that are not of interest can simply be deleted.
Messages that haven't (yet) been read, are notably
indicated. etc. Doing the same in a BBS requires
maintaining all of that state for each user *on*
the BBS server (e.g., imagine if I want to read the
first and fourth messages in a thread, delete/discard
the third and leave the second unread).

They don't have to and mosts forums display nearly all
messages in threads or only display the most recent threads
or messages in a thread by date or quantity.

Right. And everyone loves reading an entire thread
just to cherry pick the messages they want? And,
to do so again tomorrow because the BBS didn't remember
which messages they had already read or have chosen
to ignore?

Remember, this is a PRODUCT SUPPORT FORUM. People ask
specific questions about specific problems *they* are
having. They typically don't want to read everyone
else's messages just to stumble on a reply intended
for them.

There are very big differences in push and pull
technologies. And their impacts on the service
and its users. BBS's are The New News. But, they
have all the same characteristics *of* News -- just
a dressed up *remoted* interface.

You have no idea of the facts of what will be required to
maintain a SMTP type service, or the problems that can be involved.

You've misunderstood the proposal.

Remember too, many users will not want to to use their real email
address - ever.
Their email addresses will never be exposed -- that's the
whole point. If they don't wish to have any involvement,
they don't have to participate.
What about those genuine customers that want to participate but
don't ant to
give you their real email address (even if it isn't released)?
Then, what do you define as "participating"? Should we USPS mail
notices of updates to you? Are you willing to provide your
name and STREET address? Are you willing to pay for a CD
to be shipped to you with any software updates on it?

That is what a lot of users expect even these days as a CD is
easier to reload, when drives crash or systems are upgraded.

Great! Send us $X and we'll send you a new software
update. If you have any questions, you can call the
FOR PAY tech support hotline at $Y per call. We
won't even ask you your name! As long as your cash
is green...

And, if you want that update in less than the 3 days
that it typically takes the USPS to deliver it
(assuming we put it in our outbound mail RIGHT NOW),
you can pay the $20 for overnight delivery -- unless,
of course, it's Friday or a holiday...

Be *very* happy to accommodate users like this! These
activities are easily outsourced and require no real
knowledge of the product. Everything is a script
handled in a call center someplace.

The appeal of user forums is that you talk to people who
actually *use* the devices. And, they tend to speak
more "like you" than like an egghead, etc. And, aside
from those impatient know-it-alls (who will quickly be
perceived -- and censored? -- as such), most folks are
happy to try to help you with your problem. And, benefit
from hearing the solutions proposed.

How are you going to ask questions? Call Tech Support? Or,

Most customers even today expect to contact tech support not other
customers for a lot of products (washing machines to CAD software).

Yeah, and those same customers get upset when "support"
charges them for the call! There is never a problem
supporting folks on the phone. It seems that *users* are
the ones who look for other ways of getting support as
they don't want to pay for it! (or, aren't happy with the
answers that tech support gave them!)

[Years ago, I called Nakamichi to complain of a bug in
their Dragon cassette deck. You can repeatably get the
tape counter to "count the wrong way". I own two of
them so I know its a design flaw and not some "fluke".
Their tech support folks told me "that's the way it
is *supposed* to work". "Um, no. That may be the
way it *does* work but that is not the way it *should*
work!" Had their been customer support forums back
then, I might have received a better explanation of
the problem -- and, possibly, some support for getting
it *fixed* (since others would be able to confirm
how widespread it was or wasn't)]

No free lunch. You pay in the product cost or you pay
for the support service. Personally, I don't think it is
fair to charge *everyone* for the cost of support (by
embedding it in the product's cost) when not everyone
needs it. Especially for repeat customers who already
*know* the answers (why penalize a guy who really likes
your product and buys more than one??)

[When I bought my first copy of AutoCAD, I cut a deal
with the vendor by agreeing to *no* support. He was
a bit miffed when I came knocking a few days later
with a "problem" -- thinking I had renegged on my
end of the agreement. Until, of course, I told *him*
to try to do what I had done -- sompletely legitimate.
"Hmmm... looks like a bug in the 3D modeling package!"
"Gee, isn't that what I *said* when I came in?" :> ]

It seems pretty commonplace, nowadays, to set up a
"peer support" forum. This keeps the cost of support
low(er). It also gives folks a chance to be fuzzier
in describing their problems -- instead of trying to
come up with a good description *before* they picked
up the phone to call the Support Desk. And, it gives
folks a way to share tricks that they have discovered
that may or may not have been originally intended in
the product's design. For savvy vendors, it also
gives them an ear into what their customers are
saying about their product, quality, service, etc.
As well as a free source of ideas for enhancements
and other products!

But, this sort of thing isn't without cost to the vendor.
And, anything that the vendor does that looks -- or, can
*possibly* look -- heavy-handed is very visible. E.g.,
booting someone off a forum can always lead to grumbling
as to *why* they were booted -- in the form of gossip
("He was kicked off because he said the product was a
piece of sh*t"; "She was kicked off because she uncovered
an embarassing bug"; "He was kicked off because he said
the competitor's product is cheaper/better", etc.)

This is all the more reason to make it so that the users
"self-police" instead of having a "company moderator"
present.

should we provide you with a list of all of the telephone
numbers of every other customer just so you can phone each
of them to ask your questions? Should they be required
to send transcripts of each question *they* pose in the
forum to you via USPS mail?

Most users of products rarely have the time or inclination to
do that and will do so ONLY when they have a problem.

Their life (social or working) does not revolve around one
product or its fora. Your working life may revolve around
the product because that is your job, it isn't the customers.

It's *not* my job. My job is to come up with a means
whereby customers can get this sort of information
without it costing the company much; without it
deteriorating into something as unstructured as
USENET; and *with* a sense of ownership among the
participants (i.e., by giving them some control
over it as *their* forum)

If you poke around some of the BBS's and other support
forums, you'll find that many users tend to be "regulars".
Clearly they identify with the product(s) to some high
degree -- enough so that they are willing to give their
time to sorting out other folks's problems all the while
being "uncompensated". If it were not for this desire
(of some), these forums would not be effective and as
pervasive as they are.

N.B. D.H.Pink's _Drive_ is excellent reading in this regard!

OTOH, they have to "participate" even if they only want
*read* access ("membership" is given to customers only).
Trivial to do with BBS forums, you can make them membership only.
Sure, old technology. Modern BBS's are an outgrowth of much
that originated in mailing lists.

Which you are trying to regenerate with an SMTP service but fail
to see it.

On the contrary, I see a mailing-list-on-steroids as
an effective solution to this problem. I just don't plan
on implementing it the way you *think* I would. :

 

[D Yuniskis]

[This is incredibly long. Yet, still a *terse* description
of the mechanics behind mail delivery. I think it makes
it clear how easy it is to exploit these protocols to
implement a "smart agent" that acts as a "forum" for users.]

**PLEASE** snip liberally in any followups. And, unless I
have said something GLARINGLY WRONG, skip the nit picking as
I probably won't reply. It's not meant to be *authoritative*
but, rather, *informative*! I'd much rather the effort be
directed to the questions I originally posed -- instead of
rehashing trivial details. "Assume I am competent" :>
(and don't even *bother* counting my spelling errors!)

[See the separate post "FORUM DETAILS" to skip this
MUA/MTA description.]

------------------------

When you send mail, your MUA (outlook, tbird, etc.) formats
the information that you provide for that message (intended
recipient, sender, timestamp, stated subject, message body)
and passes it off to an MTA (sendmail, postfix, etc.) for
transmission.

Most end user clients do this by opening an SMTP session to
their ("local") mail *relay*. This is "some host", somewhere.
Typically, it is named Mail.whatever or SMTP.whatever. The
SMTP service is usually (though not always!) hosted on the
"well-known port" 25.

So, when sending a message to someone@someplace, your mail
client (your MUA) introduces itself to *your* SMTP relay
(server) and "addresses the envelope" for the message:

SERVER 220 mail.isp.com (other cruft)
CLIENT HELO <FQDN of sender>
SERVER 250 mail.isp.com Hello <FQDN of sender> there!
CLIENT MAIL From:<myname@here>
SERVER 250 <myname@wherever>... Sender ok
CLIENT RCPT To:<someone@someplace>
SERVER 250 <someone@someplace>... Recipient ok

Once the envelope has been addressed, it can be "stuffed":

CLIENT DATA
SERVER 354 Enter message; end with '.' on a line by itself
CLIENT [this is the contents of the message: beginning
CLIENT with any headers, followed by a blank line -- to
CLIENT delimit the headers -- and then the message "body"]
CLIENT .
SERVER 250 Mail accepted

Once the message has been passed to the mail relay, the
session can be ended:

CLIENT QUIT
SERVER 221 mail.isp.com delivering mail

(SMTP prefaces each reply with a three digit numeric code.
This allows the client software to efficiently determine the
meaning of the message without having to parse a bunch
of text. SMTP sees lots of activity! :> )

Your mail relay will queue your message and assume the
responsibility of passing it along to its intended recipient.
This allows your MUA to *disconnect* from the SMTP server
(and from the network, in general!) without having to wait
for the message to be delivered.

Your mail relay examines the message's intended recipient.
From this, it will extract the domain in which the name
(mailbox/account) resides. It will lookup the DNS records
for that domain and extract the MX records for that domain.
Using these (in a priority specified therein), it will
attempt to contact a suitable "Mail eXchanger" for your
recipient's domain.

Your mail relay will open an SMTP session with that mail
exchanger in much the same way of this session between your
MUA and MTA. When/if that mail exchanger accepts the
message, your mail relay can consider it "sent" and remove
it from its workload (queue).

That mail exchanger may, in turn, pass the message on to
*another* server. Etc. In each case, the work of delivering
the message is being OFFLOADED to some other machine! *Your*
mail client has long since quit -- your "PC" (or whatever)
can be powered off, dismantled, sold for scrap, etc. :>

[Of course, your MUA could also choose to act as an MTA and do
all of this work on your behalf. I.e., contact the *recipient's*
mail server and deliver the mail directly to that destination.
But, that assumes those servers will welcome connections from
you (how do they know you aren't some anonymous spam host?)
*and* that you want to "be persistent" for the time it takes
to get your message delivered (which, depending on how busy
the network is and the availability of the various MTA's and
servers "along the way", could take some time! E.g., mail
queues typically time out after a *day* or more -- are you
willing to keep your MUA up for that period?)

The MX records for SomeCompany.com cause incoming mail for
SomeCompany.com to be received at "some host" -- call it
mail.SomeCompany.com (remembering this can be *any* host
any *where* -- even at SomeOtherCompany.com!). Note that
it is possible for several hosts to be performing this
function (you can have multiple MX host records of varying
"preference" levels).

Mail.SomeCompany.com has to be accessible to *incoming* SMTP
connections "at all times" (well, at least often enough that
impatient MTA's don't prematurely give up on delivering mail
to that host). Folks sending mail to <someone>@SomeCompany.com
have that mail *eventually* delivered to the mail.SomeCompany.com
host. Note that *within* the SomeCompany.com domain, you may
have other subdomains -- each with their own MX records. So,
US.SomeCompany.com may receive its incoming mail on a different
physical host than Asia.SomeCoumpany.com or Europe.SomeCompany.com.
It doesn't change the model, just the details of mail delivery
for any particular email address. I.e. webmaster@Asia may
be a different account than webmaster@europe.

Mail.SomeCompany.com buffers incoming mail from the outside
world for "some period of time". This is configurable in each
mail subsystem.

From the "eventual recipient's" point of view, the mail system
looks slightly different. The way most home access is arranged,
the end user sees incoming mail as coming *from* a POP3 server
(there are other ways of doing this that differ only in the
details and protocols involved). E.g., the end user retrieves
incoming mail from the POP3.SomeCompany.com host (for example).
Note that this can be different from the host that received the
mail from the outside world (i.e., Mail.SomeCompany.com). Or,
it could be another alias for that same physical host.

(Using POP3, ) the end user retrieves his mail via a POP3 session
with that host. This begins by the user's MUA opening a connection
to the POP3 service (usually via the "well known" port 110 -- but
that can be varied). The POP3 server greets the client and then
expects it to identify itself (the account/mailbox of interest).
For the purpose of example, assume the account is named "forum":

SERVER +OK Now what? <1234.567324@pop3.somecompany.com>
CLIENT USER forust
SERVER -ERR Lose, lose
CLIENT USER forum
SERVER +OK Win, Win
CLIENT PASS Open Sesame
SERVER +OK Abandon all hope ye who enter here

(if not obvious, "+OK" and "-ERR" are PASS and FAIL indicators.
Often the text following is ignored. In some cases, it contains
actual data)

[There is another mechanism -- APOP -- which eliminates the need
to pass passwords in the clear yet still provides authentication.]

Once the user/account/mbox has been authenticated, the client can
query the server for the contents of the mbox -- as well as alter
those. Usually, the client checks for any "nondeleted" messages
in the mbox. It may, then, want to get a feel for the nature of
those messages:

CLIENT STAT
SERVER +OK 5 295519
CLIENT LIST
SERVER +OK here we go...
SERVER 1 87
SERVER 2 16882
SERVER 3 1865
SERVER 4 275692
SERVER 5 993
SERVER .

This tells the client the number of messages in the MBOX along with
the total size of that MBOX. Then, the sizes of each message are
listed (in bytes/octets).

While this doesn't tell the client about the semantic content of
each message, it does, still, provide useful information.

For example, if there are *no* messages in the MBOX, then the client
might as well terminate the session! OTOH, if the *size* of the
mailbox starts to look excessive (perhaps the mbox has a limit on
its maximum size -- exceeding that could cause further incoming mail
to be refused!), the client might want to investigate further -- are
there "lots of messages"? Or, maybe just some *huge* message that
is eating up an extraordinary amount of resources??

If, for example, you aren't expecting/accepting "large attachments",
then the 275692 bytes (long, even for *my* ramblings! :> ) in
message "4" probably are cruft that you might want to unconditionally
delete -- without even bothering to pull all of those bytes into the
client for examination! Likewise, some diminuitive message might
just be a meaningless "terd" with NO REAL INFORMATION CONTENT (i.e.,
someone hitting SEND before typing anything in the body of the message):

CLIENT DELE 4
SERVER +OK
CLIENT DELE 1
SERVER +OK

(Note that if you delete all of the messages, there is no reason to
leave the session alive. *New* mail will not be reported so, unless
you "undelete" the messages *marked* for deletion by resetting the
session, there is nothing left to be done.) The messages are not
actually deleted until the session is closed.

Also, knowing just the sizes of individual messages can affect the
order in which you examine (process!) them -- getting the longer
messages off the POP3 server first might have some advantage in
making resources on that server more available. (don't fall into
the trap of having preconceived notions of what that server's
hardware complement *might* -- or might NOT -- be!)

The client can peek into any particular message to get a better
feel for its actual origination and content without having to fetch
the entire message.

CLIENT TOP 3 0
SERVER +OK
SERVER [the various headers from message #3 are
SERVER presented, here -- one per line. After
SERVER the last header, a single blank line is
SERVER emitted -- the "blank line" that separates
SERVER the headers from the message body!]
SERVER
SERVER .

This allows the client to determine who the message is from, its
stated subject matter, time, routing information, etc. If, for
example, the client doesn't recognize the *sender* of the message,
the client can chose to delete the message without examining its
content.

Optionally, the client can ask to peek at the first "N" lines of
a particular message IN ADDITION to the headers (e.g., by issuing
"TOP 3 N" in this example). This is useful if a mail client
wants to give a message preview to a user without downloading
the entire message body.

Eventually, some number of messages may be deemed worthy of
further examination. These can then be retrieved complete
with their headers (via the "RETR" command).

Once retrieved, they can be deleted from the POP3 server *or*
left to remain there, intact. A subsequent POP3 session will
again list them as present. The client will have to assume
responsibility for remembering which messages have been examined
and which are "new" (the "UIDL" command helps by providing a
unique SESSION INDEPENDANT identifier for each message in the
MBOX).

Once the client is finished, the session can be closed:

CLIENT QUIT
SERVER +OK Come back soon! I miss you already!!

At that time, any messages marked for deletion are, in fact
deleted. Any *new* messages that have arrived while the session
was open will be visible on subsequent sessions.

IMAP4 provides similar (but expanded) functionality.
Pick one (or both).

You can manually play with *your* SMTP and POP3 servers by
telnet'ing to them and typing these commands on the keyboard
and watching the results. Fun stuff! ;-)

The important thing, here, is to see how your MUA is
interacting with these servers ON YOUR BEHALF. And, to
realize that you could just as easily write a piece of code
to send these same commands in whatever order you deem fit
and for whatever *purpose* -- and to process the resulting
data in whatever way you seem appropriate!! (hint, hint)

 

[D Yuniskis]

OK, to create a mail based "forum agent", consider:

Create an email account called forum@SomeCompany.com.
Publicize this to those folks intended/eligible to
participate. They will direct comments for that forum
to this email address using any email client (MUA)
that they happen to prefer -- or, have access to.

Eventually, their incoming (wrt The Forum) messages
will make their way to POP3.SomeCompany.com. At some
time (periodically?), your "forum agent" (which can
be a PC, network appliance, co-lo'ed server, PDA!,
cell phone!, etc.) contacts that server using POP3.
The connection can be via a wired or wireless link.
It can be high speed or low speed (bandwidth).

The "forum agent" opens a session with the POP3
server using the protocol as outlined above. If there
is no pending mail, it can close the session, tear down
the network connection, embezzle funds, fly to Miami,
power itself off, etc.

Eventually, there will be messages pending. It can
examine these messages in increasing level of detail
(e.g., quantity, size, headers, body). Any that fail to
meet its criteria (for that detail level) as "valid
messages" can be summarily deleted. (whether you *want*
to do this or not is up to you. The agent could bounce
the message, send a polite reply, etc. Or, it could
economize on bandwidth and processing power and simply
drop the message entirely -- sight unseen. "Sender
didn't play by the rules... no joy.").

For example, your forum probably won't want to forward
1MB messages to all of your subscribers (part of the
"by laws" of your forum). So, if the agent encounters
one, it can safely delete it (at the risk of offending
the *misbehaving* sender).

Likewise, if an examination of a messages header don't
confirm the sender as eligible to participate in your
forum (e.g., spammer), the agent can drop the message
(or, perhaps, harvest their return address and send
a polite reply "This forum is open by invitation only").

Once the agent has decided that a particular message is
"legitimate", it can retrieve the message (headers + body).
It can now sanitize the headers in the message (i.e.,
removing any that are unimportant as well as the name
and email address of the original sender) and "forward"
the message to the members of the forum.

*After* this has been passed to the "outbound mail" queue,
the message can be deleted from the POP3 server. The agent
can leverage the POP3's capabilities to store messages
*and* dispose of them (unread) without taking on this duty
itself -- storage, processing power, bandwidth and *time*
saved! Your agent appears more capable than it really is!

Each message can be processed in the same manner until
all messages have been processed. Then, the agent
terminates the session and goes back to sleep, lounges
in the sun at poolside, sips martinis, etc. -- until
the next time *it* (important!) decides to "check the mail".

Mail that is to be "sent" from the "forum agent" is
processed similarly -- by another thread, by a separate
process, by a *sequential* process, etc.

Each outgoing message is passed to the account's
*outgoing* mail relay (i.e., this aspect of the agent need
not deliver mail directly to the SMTP hosts for each of
the recipients in the "membership list". Instead, it
can LEVERAGE THE ABILITIES of other hosts to do this
on its behalf -- namely, the mail relay associated with
that account -- or, contracted for that purpose).

For each outgoing message, the agent builds a list of
intended recipients and passes that in the Bcc header.
E.g., it could bundle all gmail.com recipients in
one message, all aol.com in another, etc. Or, it could
throw them all together and let the mail relay sort
them out. (there is a potential for information leak
here if the MTA's don't properly strip Bcc's from
messages before delivery -- i.e., if you ran your own
server, you could potentially *see* all recipients Bcc'd
on a message that you received.)

DoS attacks are possible -- *if* the connection to the
forum agent is visible to the outside world. But, it
doesn't *have* to be visible!! (e.g., you could sit
a PC on a desktop in your office at work connected to
your company's network *behind* their firewall and
still implement this forum agent/device using an email
address "at" your company. Your agent need only converse
with your company's POP3 and SMTP servers JUST LIKE
YOU WOULD IF YOU WERE TRYING TO ACCESS YOUR MAIL FROM
YOUR OFFICE PC!).

By contrast, an HTTP-hosted forum must, by definition,
be "visible". And, its host is typically vulnerable to
attacks from a variety of exploits related to its reliance
on "extensions" to the HTT protocol as well as indirectly
from exploits of other services present on that host
(for accounting, maintenance, etc.)

It is much easier to secure the device (forum agent) as
it has limited exposure and functionality (i.e., putting
all that information on a third-party's server means
you *really* have to hope that organization has great
security as you are now exposing your customers to
*their* potential ineptitude!). E.g., the "database"
could reside on nonvolatile media (CD-ROM) so that it
can't be altered. The application itself could run
out of masked ROM... You *aren't* going to see this
with an HTTP-based service -- that service requires
much state to be preserved *on* the server (i.e., the
contents of all the messages, etc.)!

This sort of approach has a lot going for it.
- the interface is intuitive and familiar. Everything is
a message. If a subscriber wants to ignore it, they can.
If they want to reply, they reply just like they would to
any other email message (the forum agent rewrites the
headers so the reply comes to the forum email address and
not the original author of the message). If a subscriber
wants to watch a thread, they do so locally (i.e., that
state is tracked on the client side instead of the server).
If the subscriber wants to ignore all messages from
"USER127", he can "create filter from message" and have
those automatically deleted, flags, etc.
- the subscriber has greater control over message authoring
tools. No need to rely on something provided by the
remote "hosting application".
- the subscriber can view and reply to messages "off-line".
Fetch your mail. Hop on airplane. Review messages. Type
replies. "Upload" replies next time you connect to your
mail server.
- the processing involved is trivial. The agent is just
processing strings. No fancy math. No tight RT deadlines,
etc. You're just doing pattern matching, some string
substitution and copying.
- the connection bandwidth can be greatly reduced because
*you* control how quickly posts/messages are processed
(instead of being available at the whim of users). All
you have to do is ensure that you can process incoming
messages as fast as your "elastic store" will accommodate.
- "the box" can be stripped down to *just* provide this
service (which reduces its resource requirements and
power consumption -- can you host a BBS on your cell
phone?). It need not be a "general purpose PC". It
need not run a bloated OS. It need not have gobs of
memory (volatile or nonvolatile) or a hyperthreading
processor. It can be rugged. It can be small.
- it is isolated from The Internet in all but the well-defined
POP3 (for incoming) and SMTP (for outgoing) protocols.
- *it* doesn't have to have the complexity or extensibility
of an SMTP server or an HTTP server. It has a narrowly
defined scope. It is not complex (using the definition
"anything that can't be ENTIRELY and EXHAUSTIVELY contained
in one brain" :> ) This contributes to its reliability.
- *it* controls all connections -- no passive opens
- the number of active connections that must be supported
can be reduced to "one or two" (e.g., you could operate
in a simplex mode -- contact POP3 server, fetch *one*
message, close connection; rewrite it; open SMTP session,
pass message to SMTP relay, close session; return to POP3
server for *next* message; lather, rinse, repeat.)
- the amount of memory required can be greatly reduced
since you only need to be able to handle (degenerate
case) *one* message at a time. You are exploiting the
capabilities of your POP3 server to hold pending incoming
messages for you. You are exploiting the capabilitites
of your SMTP server to deliver "processed" messages on
your behalf! I.e., you are LEVERAGING other machine's
abilities to make it appear as if you had those yourself.
(by contrast, you can't "defer" an HTTP service without
annoying your user's -- "BBS busy. Please call back
later") You can conceivably process 1MB messages with
just <1K of buffer RAM (think about it! :> )
- each incoming message is processed exactly once. If a
subscriber wants to reread a message, that is done
locally via the subscriber's email client (possibly in
concert with their POP3 host -- if they opt to leave
messages on their server). By contrast, each time a
subscriber views a message on an HTTP-hosted service,
the server must resupply the contents of the message
(and, often, lots of other cruft that acts as decoration)
- if you want to archive messages, this can be done by simply
"subscribing" an address (on another host!) that just
gathers incoming messages in it's "inbox" (if you want to
make that archive available, you can do so *there* or
let the "forum agent" fetch archived articles on request
and distribute them to particular subscribers)
- it can host "sensitive data". E.g., you can have an RDBMS
running on the box (to, for example, store all of the email
addresses, "aliases/handles" for those users, purchase
history, product serial numbers, firmware revision numbers,
etc.) without fear of its data being compromised by
something "getting root" (through some *other* vulnerable
service running on that host). Otherwise, you would have
to maintain this on an independant host. Having this
information here means you can add smarts to the device:
"Notify all subscribers who are running version X firmare
that there is a potential bug... and that an update is
forthcoming"
- because it just "looks like a person transcribing messages",
you can put it anyplace such a "person" might be encountered
WITHOUT REQUIRING EXTRAORDINARY MEASURES! E.g., you can
develop a pilot production run of 100 Model XYZ Widgets
and distribute them to 100 beta testers. You could set
up a "forum agent" running on a PC in your office connected
to your company's network *without* requiring your IT
department (BofH) to consent to hosting such software on
their "golden" servers! Or, just set up a disposable email
account "somewhere" and arrange a means for the "forum
agent" to be able to periodically contact that email account.
Later, when the product enters formal production, the forum
agent's functionality can be migrated into a bigger host
(maintained by your IT department). OTOH, if you wanted to
use an HTTP-hosted forum, then you need to convince your IT
department to drill a hole through their firewall *just* for
your box -- so that it can be accessed "from outside" (can
you spell "SECURITY RISK"?). Or, you have to convince them
to set up -- and maintain -- that forum *for* you! (IT
departments are always *so* COOPERATIVE, right?? :-/ )
- the implementation scales *down* equally well. Small
"social groups" could use it (deployed on cell phones)
to participate in "group discussions". Unlike reliance
on "social networking" sites where content is out of
your control, this allows small "cellular (not in reference
to "cell phone" but, rather, to "like a cell") sites" to pop
up with little or no overhead/expense.
- it takes nothing to administer! Aside from the subscriber
list, it need not have *any* state. Turn it off. Turn
it on. Nothing gets corrupted -- EVEN POP3 AND SMTP
CONNECTIONS THAT MAY HAVE BEEN ACTIVE AT THE TIME! It's
design relies on state being preserved externally.
- it can be moved *instantly*. Specify a forwarding address
in your email account. You can do this *without* having
to contact a "hosting service" and asking them to point
your DNS records at a different HTTP server, etc.

It should be obvious how such an agent can be designed.
The tricky parts come in when you try to do things like
adding "private" communication channels, "file area",
"message archive", etc. All are do-able. But, they
have consequences.

E.g., the easiest way to handle private communications
WITHOUT INCONVENIENCING USERS is to set up a domain
"forum.somecompany.com". Mail to *the* forum is addressed
to "forum@forum.somecompany.com". It is processed as
described above. All mail to forum.somecompany.com is
routed to the "forum agent".

Each user is given an email address at forum.somecompany.com.
The address masks their actual email address and acts
just as an alias. Incoming mail from johndoe@someplace
(after verifying that johndoe@someplace is a valid subscriber)
is rewritten to, instead, appear *as if* authored by
user1285@forum.somecompany.com. (In actual fact, the
namespace probably should be very sparse to minimize the
chances of someone *on* the subscriber list addressing a
message to an address without *knowing* it to be valid).

*NO* mail is accepted at forum.somecompany.com unless the
sender is found to be a subscriber. I.e., a NON-subscriber
can't send mail to "user<N>@forum.somecompany.com" (where
N is sequentially varied from 1 to ABigNumber).

Mail sent from the forum agent always carries a Reply-To header
of "forum@forum.somecompany.com". The "From" header contains
the alias of the actual sender (e.g., "user1285@...").
A recipient can reply to a message "normally" which causes
the reply to be distributed to the entire forum.

However, the recipient can also elect to override the
reply address and, instead, specify the alias of the
original sender. In either case, the reply comes back to
the forum agent where it is validated ("is the sender
allowed to address messages to folks in this domain?").
Then, it is routed -- either to the "forum software"
for processing and redistribution; or, to the party
named by the alias.

In the latter case, the reply is routed directly *just* to
the original poster. Replies to *that* are routed just to
the original "replier". A private conversation can thus
ensue. At any time, the conversation can be "reintroduced"
to the subscribership at large by simply including
"forum@forum.somecompany.com" in the Cc line.

An alternative is to embed the addressing in the body of
the message. This gets clumsy and requires lots of
effort on the part of the subscriber. (if you have
control over somecompany.com's SMTP server, you can
hack the SMTP rewriting rules to extract these
"bogus" addresses from the RCPT field and rewrite the
message body on behalf of the subscriber. But, if you
are going to do that, you could just as easily install
these as aliases on that server!)

Regardless, the question remains as to the desirability
of providing these private channels. While I think
they are invaluable, they can also deteriorate into
"unrelated use" -- including criminal use (!)

With an entire mail domain at your disposal (namespace),
you can support multiple forums -- each at a different
virtual host (or, each at a different "email address").
For example, anotherforum@forum.somecompany.com or
forum@anotherforum.somecompany.com.

You can also add other "special addresses" with special
processing. E.g., "censor@forum.somecompany.com"
would allow subscribers to *forward* (again, using the
mechanisms they already have on their email clients!)
messages that they consider off-topic, inappropriate,
etc. to this address. A process can record the identity
of the "offender", the messageID, and the identity of
the "complainant" and, "act accordingly".

Another address, "help@forum.somecompany.com" might
simply generate a canned message that offers assistance
to subscribers. "Send *anything* to that address and
you will receive a reply containing tips on how you
can use the forum -- as well as how you can get more
detailed assistance, etc." A tag line appended to
the bootom of each message SENT OUT from the forum agent
can provide this reminder!

Likewise, a subscriber can turn delivery of messages on
or off at will (with "unsubscribe@forum.somecompany.com"
and "subscribe@forum.somecompany.com").

Building on this, perhaps "files@forum.somecompany.com"
could retrieve a list of files available -- along with
instructions on how they can be retrieved (mailed to
the subscriber).

[The *toughest* interface to provide is to access forum
(message) archives. This wants to be interactive since
users often aren't sure what they are looking for -- or,
need to examine lots of "potential" messages in the
hopes of finding what they *really* want. This may just
be too orthogonal to implement.]

Other than that, this sort of approach is very friendly
to users as it is very intuitive. It requires very
little "extra" processing. But, it means you need an
entire mail domain set aside for your forum! E.g.,
you can't implement this using just a single email
address as originally described! If you try to give
individual users their own aliases, then you have to
either switch to a different protocol -- or, play
tricks "forwarding" all of those aliases to "forum"
and having *it* redispatch them to their respective
*real* email addresses.

Of course, there is nothing that says a particular forum
must implement all of these features! E.g., the code
that runs "forum@forum.somecompany.com" can operate in
the absence of the "files@forum.somecompany.com" code.

So, what other tricks could I play?

 

[Paul Carpenter]

In article <hpp51u$j94$1@speranza.aioe.org>, not.going.to.be@seen.com
says...

Hi Paul,

Paul Carpenter wrote:
SMTP driven needs quite a bit of processing on each message
to remove all sorts of problems.

Sanitizing mail is no harder than sanitizing posts on a BBS.
You only have to sanitize each incoming message *once*. I
suspect there are probably more risks to be protected against
in adding "arbitrary data" (text) to a web page than to
a mail message (? this may be an even score as many mail
clients effectively understand and respond to "stuff" in
the body of the message :< )

You will need to effectively regenerate the WHOLE set of headers
to truely sanitise, as your users will have to exist as usernames
on your SMTP domain, so quoting and looking at email headers
will completely hide them.

This makes you an anonymising service, admittedly to a smaller set
of potential readers than normal ones. This may have other implications.

A BBS (assuming we aren't talking about accessing that via an SMTP
gateway) is a pull technology. This means it has to accept
requests whenever the user wants to access it. As such, it's
hosting must be 24/7/365 (or, some other established set of
"business hours").

Which considering the costs of webhosting these days is cheap and
insignificant to set up, as even your own server with whatever on.

And we NEVER hear of hosts on the 'net being "compromised",
right? Never hear of big firms having data stolen off
their servers (gee, don't they *know* there are risks??)?

90%+ of those types of problems are bad setup, or users putting
things on their servers that should not be there or in easy
accessible areas.

Would *your* employer (or clients) let you set up a web site in
the company's domain? How much grief would the IT department
give you?? How much overhead (administrative) would
*inevitably* result?

Lots of companies do this already, and I currently administer
over 20 domains and several small company websites including
online databases. Which takes about 1-2 hours a month of my
time.

Any server that has to be available "at will" is also exposed
to hackery as it *is* accessible. Consider how much easier
it is to protect an SMTP interface than an entire *host*!

You are joking aren't you?

An SMTP interface STILL means you have to close off other
services and ensure that even DOS attacks, by flooding a
closed http, telnet or other port does not slow service.
It still has to receive packets to reject closed ports
which takes up bandwidth restricting your valid service.

I think you misunderstood my approach. There is no "exposed
server". Just a box that emulates what a human being could
otherwise do. I.e., open mail addressed directly to it,
copy body of message into a new/empty message, mail that
new message to a group of anonymized subscribers. When
all messages processed, bring the *interface* down -- you
aren't talking to anyone so why leave it "up"?

You misunderstand SMTP, email delivery and what has to be exposed.

When your miracle solution with only POP in and SMTP out has its
interfcae 'down', where does the INCOMING email go to?

The email will be on somebody's permanently on server awaiting
collection, which is just the same as having your own server.
That server is vulnerable, and probably more so because it will
be the collection point for more people than you.

When you interface is 'up' you will need a lot more services
running starting with DNS client and ports so that the email
can be sent even if your upstream permanently connected
service is going to route the email for you.

It's not a service in the classic sense of the word. It
is, instead, just an agent. It looks like a POP3 client
*and* an SMTP client. The service that it provides is
implemented within those confines.

So you are doing a mailbox orientated listserver, something
I do know about having run a few for over 10 years.

Most of which that ran without a web front end at least
for list management have closed down as 99% of the population
no matter how tech savvie, want web interfaces. You are
trying to set something up that the vast majority of potential
users do not want to learn how to use.

By contrast, unless you get a dedicated box co-lo'ed
just for you *stripped* of all "nonessential services",
chances are, that webhosting service you outsource to
has all sorts of cruft running on their server. They

Most dedicated servers can run what you like. I have seen many
services where it can be stripped to almost nothing. This allows
you to run your own dedicated email server that is not shared
in any way.

*want* to have lots to offer. And, each of those offerings
is more code that can be buggy, exploited, hacked, etc.

So could yours. Choose your hosting (even your own wisely).

Some other "virtual host" running on that server can
be compromised and now you are at risk.

Just the same as the mailserver you are going to collect
your POP3 email from most likely will be. Many mailservers
and mailservices have been compromised over the years.

OTOH, something
would have to travel up the pipe through the POP connection
(which is initiated when *my* "agent" choses) in order to
put my agent at risk. The gate is much narrower and
more well defined.

You have utopian dreams and don't understand where your INCOMING
email is going to be stored.

DoS attacks are a fact of life for *anything*. Having
a fixed presence on the 'net just makes you a *sitting*
target. OTOH, if you are just a *client* that connects
to the network unpredictably, someone has to know *where*
you are and *when* you are there. You can *move* whereas
a server sitting at a fixed symbolic address (DNS/DDNS)
or static IP can always be "found".

People even in dial up days had DoS attacks, sometimes randomly
sometimes specificaaly, as soon as you connect something is
visible about and has to be.

You could only get better protection by using a SMTP
service on dial up to do a pull, to reduce visibility.
Then again a ping attack with 64k blocks of data will slow
you down.

You don't have to resort to dialup. That just throttles
the connection. Note that my "agent" can ignore *all*
traffic until/unless it *wants* to talk. It needn't
sit on any connection forever. And, doesn't need
to have a dedicated IP. It can look just like a mail
client -- that fetches mail directed *to* it and sends
out mail *from* it.

The upstream server for your POP3 email collection has to be permanently
on and is vulnerable to all of the same attacks and problems, you
just put it one step further down the chain.

Traffic to the "server" servicing the "group" can be
controlled by the server itself (i.e., *it* can decide when to
go fetch its incoming mail and when it wants to *send*
its outgoing mail).

Only if it is on Dial up, SMTP sits there waiting for messages
just like HTTP servers sit there waiting for web requests.

You only need SMTP on the box *to* which the agent connects
(for outbound mail delivery). The "forum" itself need not
reside/execute on that machine. Just it's "published address"
(i.e., the email address to which users direct their posts).

Where do you think that email the users send will reside?

It will reside on a permanently connected server one step
away from you for POP3 collection.

I think you do not understand differences between sending email
from a client using SMTP and the receiving end using SMTP, it
has limited choice on receiving email. You would need somebody
else's server to receive it and then fetch it from there like
POP/IMAP works for email CLIENTS.

Exactly!

The "forum agent" connects to an (inbound) mail server using POP
(or IMAP) to "fetch all mail addressed to FORUM@SOMEDOMAIN". It
"reads" those messages (from folks with real email addresses).

Where does it read the email from but a permanently connected
server, which is vulnerable, and your running a list makes
it more vulnerable. For a start most services have mailbox size
limits, which you could end up exceeding due to high spam levels
or users attaching 5MB images or other things to. These then
fill the mailbox before you 'collect' them and the upstream service
is bouncing other user emails, because the mailbox is full.

....

The forum agent need not store messages (unless you wanted to
maintain an archive) or any state regarding who has read
each message.

Depending on country of location and major countries being dealt with
you may need to store the messages for various legislations covering
security and anti-terrorism legislation. Regardless of what you
might think you may be forced by law to maintain that copy for
anywhere from 7 to 10 years.

Mainly because of the anonymising nature of your service.

.....

Do you realise how many scripts you are going to have to run to
sanitise the emails, to remove unwanted types of headers like
Reply Receipt and read receipts, let alone attachment, character
set, virus/malware scanning and other isues.

You *don't* have to do that on a BBS? C'mon, who are you
kidding? Content is content. If you have to sanitize it in
mail, then you have to sanitize it to at least the same
extent in a web based presentation!

You have to do it more in email environmentas you have LESS
control over the content type of source. Having seen emails
that have just been encoded messages that in turn actually
contain the message in different attachment formats plus
any other attachments.

On a BBS or web based interface you only have one format
type and a defined limit of attachments to deal with.

In email I have seen messages sent in up to three format
versions in the same email, and you have no control on
the number of attachments, or the order of any of the
above sections.

If your HTTP based
forum allows attachments, then they have to be sanitized
just the same. If you disallow attachments there, then,
to be fair, you can choose to disallow them in a mail-based
implementation.

In most cases banning attachments in email means for a lot of
users their emails are stripped as the HTML or RTF format
version of their email IS an attachment, whilst the plain text
version just says "This message is in Rich text format".

.....

What format are the emails to be as lots of emails get sent
as plain text and/or HTML/RTF or even ascii encoded binary only.

Having of old run mailing list for technical users who could not
drive their email software, and remote email servers around the
world misconfigured, the work involved was not trivial.

I've run several lists in the past which was what inspired
this solution. People have email, now. If it doesn't
work, it's something *they* fix -- because they depend on
it for other communications. If your HTTP based solution
breaks (virus, bug, etc.) then *you* have to fix it -- the
users are at your mercy.

If your system breaks, or gets a virus, or its upstream INCOMING
email server breaks, or your link is down, or many other scenarios
that happen regularly, you are in the same situation.

At some point in time some part of ANY solution breaks.

How many web sites are "requires browser XXXX version YYY"?
Or, "best viewed at XXXX rsolution"? Or, unfriendly to
users with disabilities? I.e., all browsers are not created
equally.

No that is how the web page is designed (VERY badly) not the
browser itself.

....

Imagine how much harder it is
to hack a "Mail Station" than a "general purpose PC"...

Sorry the vast majority of 'mail stations' are general purpose

No, I meant "mail station" as in "a network appliance
designed solely to send and receive mail". I.e., it
*just* talks POP/SMTP as a client of POP/SMTP servers.
It doesn't/can't do anything else (i.e., that's what the
"forum agent" can look like from a gazinta/cumzouta point
of view.)

Which has to run other TCP/IP services, including DNS, that has
to talk to your upstream permanently connected server.

....

PCs, maybe in a server hardware configuration, running some form
of Nix, Windows or Windows Server, using the standard network
stacks to talk to email server software like sendmail, postfix
qmail, ftgate.........

See above. If *all* you have to do is talk to *a* host
(or pair of hosts) provided by ISP/employer then you can
develop a very narrowly defined system that doesn't carry all
that extra baggage/liability. E.g., you don't need a
generic stack, you don't need to support lots of
connections, you don't need lots of horsepower, etc.
Your role is *just* to pull in messages, massage them
and spit them back out.

You are assuming a lot of utopian ideas and moving the issues
elsewhere, which will have financial, managerial, political
and other issues you are deliberately avoiding.

.....

So they have the same and MORE issues of security.
I think you are confusing SMTP as you see for sending email and
how SMTP has to work on a server, i.e. be normally available
24/7/365.

No, you have assumed that the "forum agent" has to reside
*on/in* a "mail server" and *act* as a classic "server".

NO the RECEIVING MAILBOX has to sit somehwere on somebodies
system that is on 24/7/365.

Look at my FORUM DETAILS post, elsewhere in this topic.
Hopefully, that better explains the model I am proposing.

It is a proposal that is utopian, few users would want to
use, and you are wearing blinkers.

(the former has a very narrowly defined interface whereas

Standard network interface be it ethernet or dial up via PPP, that
ends up as TCP/IP to the email server software.

You can generally ping mail stations.

the latter has to be adeptly managed to ensure it has
the smallest visible interface possible)

That is true of both 'types' of system as they are basically the
same.

Again, you've missed the model I am proposing.

You have missed the fact that you do not have obscurity or security
you are depending on somebody else or many others to do the work
'magically' for you.

If you think they are the same, consider how you would
implement my model using HTTP as the transport protocol.

There is NO transparent protocol as somewhere there has to be
a 24/7/365 email server to RECEIVE the email, you have just
moved a few points of vulnerability from you to several others.

.....

Mail *can* be delivered "almost instantly". Yet, do
you send a message (to a friend at work) and then
sit there waiting for a reply to it? If it doesn't
appear in 30 seconds, do you resend it?? Or, do you
move on to some other activity and "check back later"?

Mail is NOT a guaranteed delivery system. I have seen
many people furious someone ha not replied to emails in 15
minutes.

OTOH, when you go to a web site, you *expect* it
to be "nearly instantaneous". You grumble if the
web page loads slowly, etc.

You cut your web server and bandwidth to suit your
requirements.

Or, am I the *only* one who behaves this way?? :

No but email is not the utopian answer you think it is

And, if you are posting a product support question,
all you are going to see is your post -- until someone
else stumbles onto the (HTTP hosted) site and opts
to answer it. So, you *still* have to wait for a
reply -- even though the site is "online" and always
accessible. (you can "check your mail" for a reply
at least as easily as you can go to a web site -- even
if it is bookmarked -- and "login"; then browse to find
your post and see if there are any followups!)

Many sites have email on followup flags which is easy to maintain
as it uses next to no processing is the same as listserver.

And, it can leverage a multitude of other hosts -- the
mail hosts for each of its subscribers -- to act on its
behalf (by contrast, a pull technology has to carry all
of the load by itself).

You really do not know what you are talking about.

Defend your accusation. I have claimed that my "forum
agent" (running off a PDA in Starbucks coffee shop)
can send *a* message and that it can then rely on
dozens or hundreds of other machines around the planet
to store and forward that message, as needed, to
reach each of the individual addressees in that message.
I.e., they "act on its behalf". And, as such, "leverage
a multitude of other hosts".

The only 'multitude of hosts' sharing the task is the ISP
outgoing email server(s) that you are expecting to suddenly
distribute potentially thousands of emails. Their servers have
to split your outgoing list into separate deliverable messages.

This in turn could make you look like a spammer, will require
careful delivery of messages in batches.

Even when the messages have reached their ultimate
mail hosts (for each of the subscribers), those hosts
are *still* acting on my behalf -- by buffering the
message(s) until the users connects to their mail
hosts and retrieve their mail. My "PDA forum agent"
could be powered off and sitting in my glovebox by
that time...

The main job you are offloading is not their end but the initial
sending out.

That is why I say you have no idea.

[I use the PDA as an example as it shows how limited
the role that "forum agent" can take! I am not
advocating this method of implementation -- though
there is nothing that precludes it!]

Whether it is a PDA, network agent, PC or full blown server
your method works the same as you only obviously understand
email as a user on an ISP.

You are trying to offload a lot of the 'multitude of hosts'
work onto your ISP who may well very quickly see you as a spammer
or abusing your service agreement.

Consider that the "server" can send one copy of a message
to a multitude of recipients in a given MX. The "remote"
MX can then handle distributing those copies to the N
recipients that it services (instead of waiting for those
N recipients to come *to* the "server").

You are talking about using somebody else potentially as
a recognisable Spamhost. How to get your ISP to cut off your
service.

TI sends out how many thousands of email messages each day
telling you about there latest and greatest chips that they
are promoting "this week" (tomorrow, there will be something
equally grand and wonderful -- at least in THEIR opinion).
Are *they* a spamhost? Do you think the folks who
provide network access to TI are going to cut off their
service??

1/ They use their own or CONTRACTED email servers to do this

2/ I know that they will doing all sorts of managed things
behind the scenes to batch the emails, try to ensure
they get off of spamhost lists.

3/ Even major corporations from banks to ISPs regularly get
on one spamhost list or other every week.

OTOH, if the updates were *pushed* to
users, then MS could better manage the problem. Any
effort "waiting" on a server is waste -- especially
if you have to come back later to get what you want
(and still have no assurances of being able to do so!)

The overhead of doing that by things like SMTP is ridiculous
as the method does not support binary transmission, you have
to ASCII encode your binary, bloating it and taking longer
to send.

Ah, yes. And HTML is *so* terribly efficient! C'mon,
think about what you're saying... Everything is ASCII/UTF8.

No I know what I am saying, attachements in email sent as some
form of encoded text.

On web pages the HTML maybe text, but the images and other
similar attachments are as BINARY as long as the correct mime type
is setup and negotiated.

Plus, damn near everything has an HTML tag associated with it.
And, possibly some scripting embedded within (Gee, my browser
doesn't support this version of Java...).

That is down to bad website design, you are confusing the tools and
the content.

And, of course,
all the decorations that accompany the page (isn't google's
front page refreshingly sparse??! Of course, there's nothing
*there* of interest besides the image-of-the-day :-/ ).
Plus, all the links to other pages "just to be helpful".

So there are a lot of bad web pages out there.

There are a lot of emails sent as web pages by marketing departments
that are just the same.

And, lets not forget the inevitable ads that alwasy seem to
creep in -- even if they are for your company's products!

Also on lots of emails.

Or, are we now saying user must support HTTP for the
BBS access and FTP to pull down files? Any other
protocols you want to add?

That is what the USERS PREFER TO USE, that is from
experience.

.....

What you are trying to setup is what YOU want, not what
your POTENTIAL USERS WANT.


--
Paul Carpenter | paul@pcserviceselectronics.co.uk
<http://www.pcserviceselectronics.co.uk/> PC Services
<http://www.pcserviceselectronics.co.uk/fonts/> Timing Diagram Font
<http://www.gnuh8.org.uk/> GNU H8 - compiler & Renesas H8/H8S/H8 Tiny
<http://www.badweb.org.uk/> For those web sites you hate

 

[David Brown]

D Yuniskis wrote:
<snip>

And we NEVER hear of hosts on the 'net being "compromised",
right? Never hear of big firms having data stolen off
their servers (gee, don't they *know* there are risks??)?
Would *your* employer (or clients) let you set up a web site in
the company's domain? How much grief would the IT department
give you?? How much overhead (administrative) would
*inevitably* result?

Do you have any concept of how many servers of various sorts are
attached to the Internet at any given time? There are probably
millions. And in the huge majority of cases, they run happily without
being hacked, DOS'ed, defaced, etc.

If you want to connect /anything/ to the Internet, whether it be a
server or a client (including your proposed system), there are some
basic rules (in no particular order) to follow and you will be entirely
safe.

First, make sure that only relevant services are open to the Internet.
For a client, that means /no/ services. For a web server, it means only
port 80 - and so on. If you can't be sure that you only have the
required services open, make sure you have a hardware firewall between
you and the Internet, and only allow in the relevant traffic. If you
are brave enough to use Windows as a server, remember that you always
need an external firewall. This eliminates the risk of being attacked
through other ports and services.

Second, make sure that the server has the minimal amount of data, no
clear-text passwords, no financial or other compromising data, and
minimal access to other systems on your network. This limits your risks
if you /do/ get broken into.

Third, make sure you have strong passwords, disable root logins (thus an
attacker needs to guess user names as well as passwords), and move
remote access (such as ssh) to a non-standard port.

Fourth, do some research about the security record of the software you
are running. Keep track of security announcements for the software, and
know how to keep it updated as needed.

Fifth, don't run any unnecessary software on the server. A web server
has no need for a windowing system or application software.


In your case, you are not going to get DOS'ed - you are not big enough
to be of the slightest interest to attackers. You are not going to be
attacked by anyone serious - you have nothing worth stealing. (I say
this with complete confidence - if you /were/ big enough to be a target,
you would have hired professionals to do the job.)

I also suspect that your fears of spam are exaggerated - why would
someone bother? Spammers look for success rates in the order of one
"hit" per hundred thousand spams - there is no point in invading a
group/forum/website/etc. unless it has many thousands of regular readers.

So the only attackers will be worms and script kiddies using scanners
running through groups of ip addresses using common vulnerabilities.
The script kiddies look for ssh (only on port 22), open windows shares,
badly configured web servers, etc. Since these will all be blocked (and
your web server will be correctly configured), they are not a problem.
Similarly, most worms will also be blocked.

In the end, the only vulnerability is if there is some serious flaw in
the service software you are actually using, and a worm or script that
attacks the flaw is popular before you've updated the software.
Realistically, that's a very small risk.

Security is never absolute - it's about getting the functionality and
service you need while minimising risks and the consequences of
breaches. But securing a server isn't that hard once you know the
principles.

As a side note, the reason big firms get hacked and their data stolen is
that they have something worth stealing, and haven't followed the basic
rules. It's amazing how bad many big companies' security is.

 

[JosephKK]

On Fri, 09 Apr 2010 21:05:31 -0700, D Yuniskis <not.going.to.be@seen.com> wrote:

Hi Joseph,

JosephKK wrote:
On Wed, 07 Apr 2010 18:39:48 -0700, D Yuniskis <not.going.to.be@seen.com> wrote:

I need to design a "User's Forum" for an upcoming product release.
The goal (as always) is to let users share their ideas, solutions,
gripes, etc. with each other directly without going "through"
any business structure.

Interesting task.

frown> Annoying task. But, the product is really interesting
so its worth the effort to get it right. :-/

There are two well done webforum & NNTP
combination systems that i have seen: OpenSuse forums and
Microsoft public forums. Both are actively managed. Yahoo

By "managed", I assume you mean there is a human overseeing
their operation? (moderator, of sorts) Sorting out the
spammers, abusers, etc.?

More likely a decent sized team with software power tools to help.

forums also appear to have SMTP gateways, but they overdid
the registration thing. Registration is a good idea anyway,
it could require reading an email and using a key (good for

This is a closed list. Only folks owning the product

That is clear from all that i have read.

get on it. It isn't visible -- or accessible -- otherwise.
Granted, an "owner" could have a bot accessing it on his
behalf. But, if that bot misbehaved (however the rest of
the subscribership defines "misbehaving"), then it's
out!

The number of people capable of creating BOTs are far fewer
than those that cannot learn enough self protection to avoid
unknowingly hosting a BOT.

a year?) in order to post (anti-bot). Regular spam blocking
will also have to be considered, as well as automatic

Spam from "foreign" sources is gated by membership list.
If a subscriber starts posting spam, then other subscribers
boot him/her from the list. (does buying another instance
of the product give you another subscription??)

blocking of profanity. Take a look at STUMP robomoderation
program and group distributed moderation. For people that
wish to continue to private email they should have the wit to
take anti-harvesting precautions (don't block it, but don't

I think you just scrub the email addresses (and "real names")
from all posts. Its trivial to map each real user into
an arbitrary (though consistent) "anonymized user".
I.e., the only thing you see is the message content.
If you are observant, you might notice a trend in some
of the questions from a particular "anonymized user".
Some information could leak that way.

Perhaps be a little more active, have the user create a nym
as part of the sign up process.

Of course, there is nothing to stop folks from posting
(in the message body) their real name, email address, etc.
(You can't prevent people from shooting themselves in
the foot! :&gt

encourage it). Provide a FAQ and post some policy basics
regularly. The FAQ should contain some "how to make a good
post" information.

 

[Paul Carpenter]

In article <e5WdnQ74ZvSAfV3WnZ2dnUVZ8qydnZ2d@lyse.net>,
david.brown@hesbynett.removethisbit.no says...

D Yuniskis wrote:
snip

And we NEVER hear of hosts on the 'net being "compromised",
right? Never hear of big firms having data stolen off
their servers (gee, don't they *know* there are risks??)?
Would *your* employer (or clients) let you set up a web site in
the company's domain? How much grief would the IT department
give you?? How much overhead (administrative) would
*inevitably* result?


Do you have any concept of how many servers of various sorts are
attached to the Internet at any given time? There are probably
millions. And in the huge majority of cases, they run happily without
being hacked, DOS'ed, defaced, etc.

If you want to connect /anything/ to the Internet, whether it be a
server or a client (including your proposed system), there are some
basic rules (in no particular order) to follow and you will be entirely
safe.

... snip lots of sensible points

Security is never absolute - it's about getting the functionality and
service you need while minimising risks and the consequences of
breaches. But securing a server isn't that hard once you know the
principles.

As a side note, the reason big firms get hacked and their data stolen is
that they have something worth stealing, and haven't followed the basic
rules. It's amazing how bad many big companies' security is.

Just as bad are many government or related agencies websites, that have
lots of low paid and under-trained people sideways promoted into data
entry for the website. In one case in UK, easily findable excel
spreadsheets of personal information.

Anyway over here much easier to find laptops/pen drives on trains, in
laptop bars, taxis, or restaurants from all sorts of departments. Let
alone the amount of details they send via the post on CDs.


--
Paul Carpenter | paul@pcserviceselectronics.co.uk
<http://www.pcserviceselectronics.co.uk/> PC Services
<http://www.pcserviceselectronics.co.uk/fonts/> Timing Diagram Font
<http://www.gnuh8.org.uk/> GNU H8 - compiler & Renesas H8/H8S/H8 Tiny
<http://www.badweb.org.uk/> For those web sites you hate

 

[MooseFET]

On Apr 10, 1:31 pm, David Brown
<david.br...@hesbynett.removethisbit.no> wrote:
[....]
I'm going to add a few comments.

Second, make sure that the server has the minimal amount of data, no
clear-text passwords, no financial or other compromising data, and
minimal access to other systems on your network.  This limits your risks
if you /do/ get broken into.

Also if you have other Windows machines on the network, place another
firewall in the system so that they can't be the source of the attack.
A user can have a virus which does anything. If the user can post
your valuable information, so can a virus.

Third, make sure you have strong passwords, disable root logins (thus an
attacker needs to guess user names as well as passwords), and move
remote access (such as ssh) to a non-standard port.

Make your email names different from the login names so that
someone who has received an email won't know the login name for that
person.

[....]

Fifth, don't run any unnecessary software on the server.  A web server
has no need for a windowing system or application software.

I would have put the fifth earlier because it is so important.

In your case, you are not going to get DOS'ed - you are not big enough
to be of the slightest interest to attackers.  You are not going to be
attacked by anyone serious - you have nothing worth stealing.  (I say
this with complete confidence - if you /were/ big enough to be a target,
you would have hired professionals to do the job.)

You can get attacked just by someone trying ideas out or teenagers.
Teenagers may not know a lot but they often have lots of time to try
out different ideas.

 

[David Brown]

MooseFET wrote:

On Apr 10, 1:31 pm, David Brown
david.br...@hesbynett.removethisbit.no> wrote:
[....]
I'm going to add a few comments.

Second, make sure that the server has the minimal amount of data, no
clear-text passwords, no financial or other compromising data, and
minimal access to other systems on your network. This limits your risks
if you /do/ get broken into.

Also if you have other Windows machines on the network, place another
firewall in the system so that they can't be the source of the attack.
A user can have a virus which does anything. If the user can post
your valuable information, so can a virus.

True enough - and you should therefore never let outside windows
machines onto your network. The only time our company ever suffered
from a worm was when someone had taken a laptop from home and attached
it to our network (breaking my rules to do so). The irony is that they
wanted to use our fast internet connection to download an update to
protect against said worm.

I find that having the firewall block all outgoing smtp traffic (except
from your internal mail server to your ISP's relay), with an alert
system for attempted smtp connections, is a quick way to find most
malware and to limit its damage.

Third, make sure you have strong passwords, disable root logins (thus an
attacker needs to guess user names as well as passwords), and move
remote access (such as ssh) to a non-standard port.

Make your email names different from the login names so that
someone who has received an email won't know the login name for that
person.

That can be a good idea when you need higher security. But the majority
of attackers don't know anything about you except your ip address - they
will try common names such as "root" and "Administrator". It's a
different matter with directed attacks, in which case the attacker will
likely research some email addresses as likely login names.

Fifth, don't run any unnecessary software on the server. A web server
has no need for a windowing system or application software.

I would have put the fifth earlier because it is so important.

It is indeed important, but I didn't want to try to give an ordering.
This is only a few general points - the priority will depend on the
circumstances.

In your case, you are not going to get DOS'ed - you are not big enough
to be of the slightest interest to attackers. You are not going to be
attacked by anyone serious - you have nothing worth stealing. (I say
this with complete confidence - if you /were/ big enough to be a target,
you would have hired professionals to do the job.)

You can get attacked just by someone trying ideas out or teenagers.
Teenagers may not know a lot but they often have lots of time to try
out different ideas.

I always consider teenagers to be the most dangerous of users!

 

[Michael A. Terrell]

MooseFET wrote:

I may hold the record for the fastest infection of an XP machine. It
was a new Dell. I connected it to the network and it was a goner
before I finished getting it registered. For a short while I had two
PCs in my office somewhere I have the picture of the Dell in a
garbage can in front of the IT guys office door. That is how I
returned it to him.

If you were on a corporate network and that happened, they need to
fire the entire IT department.


--
Lead free solder is Belgium's version of 'Hold my beer and watch this!'

 

[Vladimir Vassilevsky]

Michael A. Terrell wrote:

MooseFET wrote:

I may hold the record for the fastest infection of an XP machine. It
was a new Dell. I connected it to the network and it was a goner
before I finished getting it registered. For a short while I had two
PCs in my office somewhere I have the picture of the Dell in a
garbage can in front of the IT guys office door. That is how I
returned it to him.



If you were on a corporate network and that happened, they need to
fire the entire IT department.

You should apply common sense to the horror stories that linux fans are
telling about Windows XP.

VLV

 

[Michael A. Terrell]

Vladimir Vassilevsky wrote:

Michael A. Terrell wrote:
MooseFET wrote:

I may hold the record for the fastest infection of an XP machine. It
was a new Dell. I connected it to the network and it was a goner
before I finished getting it registered. For a short while I had two
PCs in my office somewhere I have the picture of the Dell in a
garbage can in front of the IT guys office door. That is how I
returned it to him.



If you were on a corporate network and that happened, they need to
fire the entire IT department.

You should apply common sense to the horror stories that linux fans are
telling about Windows XP.

Common sense and Linux are an oxymoron.


--
Lead free solder is Belgium's version of 'Hold my beer and watch this!'

 

[MooseFET]

On Apr 11, 11:35 am, David Brown
<david.br...@hesbynett.removethisbit.no> wrote:

MooseFET wrote:
On Apr 10, 1:31 pm, David Brown
david.br...@hesbynett.removethisbit.no> wrote:
[....]

True enough - and you should therefore never let outside windows
machines onto your network.  The only time our company ever suffered
from a worm was when someone had taken a laptop from home and attached
it to our network (breaking my rules to do so).  The irony is that they
wanted to use our fast internet connection to download an update to
protect against said worm.

The last time I tried, Microsoft would not let you download the update
with
a virus free machine and then copy the file onto the down rev version
of
Windoz. You have no choice put to connect the down-rev machine to
the
outside world and let all the squirmy things get in while it tries to
get
updated to keep them out.

I may hold the record for the fastest infection of an XP machine. It
was
a new Dell. I connected it to the network and it was a goner before I
finished getting it registered. For a short while I had two PCs in
my
office somewhere I have the picture of the Dell in a garbage can in
front
of the IT guys office door. That is how I returned it to him.

I run Windows 98 in a virtual machine that has no network access. It
is a real pigs breakfast but for the few Windoz things I must do, it
is
a good option. Unfortunately you can't run XP that way because it
locks
up if it can't phone home or something.

My wife keeps here Windows machine fully updated with all the latest
virus blockers etc. Just to be safe, I reboot it in Linux from time
to
time and copy all the important information onto an external drive.
It is a good thing I've done that.


[....]

Third, make sure you have strong passwords, disable root logins (thus an
attacker needs to guess user names as well as passwords), and move
remote access (such as ssh) to a non-standard port.

Make your email names different from the login names so that
someone who has received an email won't know the login name for that
person.

That can be a good idea when you need higher security.  But the majority
of attackers don't know anything about you except your ip address - they
will try common names such as "root" and "Administrator".  It's a
different matter with directed attacks, in which case the attacker will
likely research some email addresses as likely login names.

I was thinking about an attack from someone who had a spam email
address
list. The user name and company name can be parsed out.

 

[David Brown]

On 13/04/2010 04:58, MooseFET wrote:

On Apr 11, 11:35 am, David Brown
david.br...@hesbynett.removethisbit.no> wrote:
MooseFET wrote:
On Apr 10, 1:31 pm, David Brown
david.br...@hesbynett.removethisbit.no> wrote:
[....]

True enough - and you should therefore never let outside windows
machines onto your network. The only time our company ever
suffered from a worm was when someone had taken a laptop from home
and attached it to our network (breaking my rules to do so). The
irony is that they wanted to use our fast internet connection to
download an update to protect against said worm.

The last time I tried, Microsoft would not let you download the
update with a virus free machine and then copy the file onto the down
rev version of Windoz. You have no choice put to connect the
down-rev machine to the outside world and let all the squirmy things
get in while it tries to get updated to keep them out.

It's easy to protect a windows PC from worms - you make sure there is a
proper firewall (i.e., a physical box, or a reasonably configured
Linux/BSD/etc. firewall - not a windows software "firewall") between the
PC and any potentially infected machines. So you certainly don't have
to let the worms in while getting updates. Of course, not everyone
/has/ a decent firewall, despite their very low cost (a hardware
firewall is often cheaper than a year's subscription to a useless
third-party software firewall) - that's why they've got worms in the
first place.

I may hold the record for the fastest infection of an XP machine.
It was a new Dell. I connected it to the network and it was a goner
before I finished getting it registered. For a short while I had two
PCs in my office somewhere I have the picture of the Dell in a
garbage can in front of the IT guys office door. That is how I
returned it to him.

I remember reading some numbers a while back - I think it was when SP2
for XP came out. At the time, a fresh XP (no service pack, firewall
disabled as per default) installation directly connected to the internet
had a half-life of about 15 minutes before being infected by some sort
of worm. With SP2 and the windows firewall enabled, that went up to
around 48 hours (though that figure is less accurate, since it would
take too long to get enough samples). I've certainly seem people
getting infected as they try to update.

I run Windows 98 in a virtual machine that has no network access.
It is a real pigs breakfast but for the few Windoz things I must do,
it is a good option. Unfortunately you can't run XP that way because
it locks up if it can't phone home or something.

I'd recommend W2K pro as a better choice. It runs very well within a
VirtualBox machine, and works with far more windows software than Win98.
It is also possible to install and run XP without any sort of internet
connection, but then you need to activate it over the phone to MS (or
use one of the many wpa-killers you can find in the murkier areas of the
net). But W2K is lighter on resources, and normally does just as good a
job.

My wife keeps here Windows machine fully updated with all the latest
virus blockers etc. Just to be safe, I reboot it in Linux from time
to time and copy all the important information onto an external
drive. It is a good thing I've done that.


[....]
Third, make sure you have strong passwords, disable root logins
(thus an attacker needs to guess user names as well as
passwords), and move remote access (such as ssh) to a
non-standard port.

Make your email names different from the login names so that
someone who has received an email won't know the login name for
that person.

That can be a good idea when you need higher security. But the
majority of attackers don't know anything about you except your ip
address - they will try common names such as "root" and
"Administrator". It's a different matter with directed attacks, in
which case the attacker will likely research some email addresses
as likely login names.

I was thinking about an attack from someone who had a spam email
address list. The user name and company name can be parsed out.

Again, you are safe unless someone is directly targeting you. The
automated scanners and worms scan through IP addresses - they don't make
any connection between that address and a company name. Think about it
- there is no practical way to automate the process. At best, you could
make it work for companies big enough to own IP addresses rather than
leasing them from their ISP, and in those cases you'd have too many
matching email addresses to make any use of the information. It's far
faster and easier just to move on to the next target and try common
passwords and default firewall/router logins.

 

[David Brown]

On 13/04/2010 05:30, Vladimir Vassilevsky wrote:

Michael A. Terrell wrote:
MooseFET wrote:

I may hold the record for the fastest infection of an XP machine. It
was a new Dell. I connected it to the network and it was a goner
before I finished getting it registered. For a short while I had two
PCs in my office somewhere I have the picture of the Dell in a
garbage can in front of the IT guys office door. That is how I
returned it to him.



If you were on a corporate network and that happened, they need to
fire the entire IT department.

Corporate networks are not immune to infections - there are endless
examples of big corporations and other large organisations getting
network-wide infections. But you are certainly right that if such a
network gets infected, somebody has failed to do their job.

You should apply common sense to the horror stories that linux fans are
telling about Windows XP.

It is roughly accurate to say that an unprotected XP machine connected
to the Internet will get infected by worms and/or hacked in some way.
This will happen very quickly if you don't even have the XP software
firewall enabled, and will typically take a long time if it is updated
regularly and has a software firewall enabled.

It is also roughly accurate to say that pretty much any Linux
distribution with its default installation can be connected directly to
the internet and it's very unlikely to be infected by anything during
its useful lifetime.

Of course, in either case you may install vulnerable applications or
services that significantly increase your risks.

But it is also a simple matter to protect your windows machine. Put a
$50 hardware firewall between it and the internet (or other risky
machines), and you can be confident nothing bad will come in.

There are plenty of horror stories about windows insecurity, told by
news agencies and not just linux fans:

<http://www.silicon.com/technology/security/2008/11/18/hospitals-systems-hit-by-computer-virus-39345606/>

But claims that windows can't be kept safe are often greatly
exaggerated. It is perfectly possible to keep windows machines
infection free - but you have to apply some basic rules to do so.