Re: [fw-general] Zend_Filter design proposal

[Darby Felton]

(moving thread to fw-core)

Hi Kevin,

(Sorry, this message is rather long)

Kevin McArthur wrote:
> Couple of questions -- how does this work on an input array like $_POST
> and is it iterable. A common use case is often to provide dynamic
> (client-side-generated) form elements that follow a standard name like
> description1, description2 etc... which all start with a common
> substring but have an differing ending.

This is a good question; indeed, the Zend_Filter design proposal does
not directly address how to validate incoming form data (ala $_POST,
$_GET). It is focused primarily on improving our OO definitions of what
filters and validators are, however, which, in turn, provides a better
foundation for building, for example, a form controller. Because
filtering and validation take place in various parts of an application
(e.g., input filtering and validation, escaping output), it is important
to design filters and validators such that they can be used in many
different contexts. Thus, the proposal does not attempt to enforce such
security conventions as removing raw access to $_POST data.

"Security is a process, not a product." -- Bruce Schneier

I think that the use case you describe might better be addressed by a
form controller [component], such as what is hinted by existing proposals:

Zend_Form Component Proposal
http://framework.zend.com/wiki/x/DA4

Zend_Form_Controller Proposal
http://framework.zend.com/wiki/x/mQE

It is very likely that such a component would consume Zend_Filter and
Zend_Valid.

Though we recognize the need for supporting filter and validation
operations upon /sets/ of data, it will be some time after 0.7.0 before
we can fully address this complicated problem, even if only with respect
to form processing. Consider, for example, that the current
Zend_Filter_Input class operates only upon the first-level keys of its
input array. It does not support multidimensional arrays, where the key
of interest for a particular filter or validation operation may be two
or more levels deep into the input array. I have used multidimensional
arrays via $_POST often enough that I consider this to be a common use
case, but others' experiences may likely vary.

> One of the goals of the original filtering mechanisms was to create a
> way to destroy raw post data and ensure access only through a validation
> structure. Does this not reverse this goal?

I'm not sure that destroying raw post data was one of the original goals
of the Zend_Filter component, though certainly the current
Zend_Filter_Input does this by default if you supply $_POST to it. This
probably is not such a good idea for a framework component to do,
especially since the current component alone does not fully support
being handed $_POST (e.g., try using multidimensional arrays), but
perhaps also because interoperability issues might arise. For example,
if a ZF-powered app uses libraries that depend on $_POST, there could be
problems. I suppose one could Certainly user code can trivially set
$_POST = null if it is appropriate for the use case. Maybe I'm missing
something, though; it's happened often enough before. ;)

> While you're looking at filter/validation, another useful level might be
> deterministic [figguring out if a string is an email address, or
> contains urls etc]

Indeed, these are good ideas; thanks for your feedback!

Best regards,
Darby

>
> Kevin McArthur
> ----- Original Message ----- From: "Darby Felton" <[hidden email]>
> To: "Zend Framework - General" <[hidden email]>
> Sent: Wednesday, January 10, 2007 10:34 AM
> Subject: [fw-general] Zend_Filter design proposal
>
>
>> Hi all,
>>
>> I have been working on a proposal for redesigning the functionality of
>> the current Zend_Filter component:
>>
>> http://framework.zend.com/wiki/x/Wi8
>>
>> Your review and feedback would be sincerely appreciated in this early
>> stage of development. Please feel free to post comments on the wiki page
>> or to have your say on the fw-core mailing list, as Zend_Filter is now
>> considered to be within the Core infrastructure team, and I look forward
>> to hearing from you!
>>
>> Subscribe to fw-core: [hidden email]
>>
>> Nabble fw-core archive: http://www.nabble.com/Zend-Core-f16191.html
>>
>> Thank you for your time and consideration!
>>
>> Best regards,
>> Darby
>
>

[Kevin McArthur-2]

> Though we recognize the need for supporting filter and validation
> operations upon /sets/ of data, it will be some time after 0.7.0 before
> we can fully address this complicated problem, even if only with respect
> to form processing. Consider, for example, that the current
> Zend_Filter_Input class operates only upon the first-level keys of its
> input array. It does not support multidimensional arrays, where the key
> of interest for a particular filter or validation operation may be two
> or more levels deep into the input array. I have used multidimensional
> arrays via $_POST often enough that I consider this to be a common use
> case, but others' experiences may likely vary.

I have successfully used Zend_Filter_Input with recursive structures using
SPL iterator patterns. Creating a RecurisveIterator for this is quite
simple. When I submitted the iterator patch to the old system (which was
rejected due to this proposal being in the works) it allowed for treating
the collection as an Iterator --- which made it easy to integrate with SPL
recursive iterator patterns. So yes, this was sort-of supported.

> I'm not sure that destroying raw post data was one of the original goals
> of the Zend_Filter component, though certainly the current
> Zend_Filter_Input does this by default if you supply $_POST to it. This
> probably is not such a good idea for a framework component to do,
> especially since the current component alone does not fully support
> being handed $_POST (e.g., try using multidimensional arrays), but
> perhaps also because interoperability issues might arise. For example,
> if a ZF-powered app uses libraries that depend on $_POST, there could be
> problems. I suppose one could Certainly user code can trivially set
> $_POST = null if it is appropriate for the use case. Maybe I'm missing
> something, though; it's happened often enough before. ;)

The idea is more of social enforcement of good coding practices -- highly
useful for team development. Direct access to post data means that it will
probably be used and this can cause some issues that I thought ZFI solved.

Kevin

>
>> While you're looking at filter/validation, another useful level might be
>> deterministic [figguring out if a string is an email address, or
>> contains urls etc]
>
> Indeed, these are good ideas; thanks for your feedback!
>
> Best regards,
> Darby
>
>>
>> Kevin McArthur
>> ----- Original Message ----- From: "Darby Felton" <[hidden email]>
>> To: "Zend Framework - General" <[hidden email]>
>> Sent: Wednesday, January 10, 2007 10:34 AM
>> Subject: [fw-general] Zend_Filter design proposal
>>
>>
>>> Hi all,
>>>
>>> I have been working on a proposal for redesigning the functionality of
>>> the current Zend_Filter component:
>>>
>>> http://framework.zend.com/wiki/x/Wi8
>>>
>>> Your review and feedback would be sincerely appreciated in this early
>>> stage of development. Please feel free to post comments on the wiki page
>>> or to have your say on the fw-core mailing list, as Zend_Filter is now
>>> considered to be within the Core infrastructure team, and I look forward
>>> to hearing from you!
>>>
>>> Subscribe to fw-core: [hidden email]
>>>
>>> Nabble fw-core archive: http://www.nabble.com/Zend-Core-f16191.html
>>>
>>> Thank you for your time and consideration!
>>>
>>> Best regards,
>>> Darby
>>
>>

[Darby Felton]

Hi Kevin,

Kevin McArthur wrote:
> I have successfully used Zend_Filter_Input with recursive structures
> using SPL iterator patterns. Creating a RecurisveIterator for this is
> quite simple. When I submitted the iterator patch to the old system
> (which was rejected due to this proposal being in the works) it allowed
> for treating the collection as an Iterator --- which made it easy to
> integrate with SPL recursive iterator patterns. So yes, this was sort-of
> supported.

This makes a lot of sense to me. We know that we will need something
that integrates running multiple filters and validators upon such data
types, though with respect to the current design proposal, I would
consider it highly related but perhaps out of scope. If this idea can
hold, then it gives us the opportunity to propose such functionality
separately, and I would be happy to facilitate your ideas on a design
that would be appropriate for the framework. Maybe you would like to
seed and/or write a proposal to illustrate how we could do this?

What might others be thinking?

> The idea is more of social enforcement of good coding practices --
> highly useful for team development. Direct access to post data means
> that it will probably be used and this can cause some issues that I
> thought ZFI solved.

Yes, I see your point, and we may want to continue to offer this feature
with the framework. I didn't really address this topic with my design
proposal, but I think it deserves an "official" response. The questions
to answer might be:

Do we wish to provide the ability to set $_POST to null, beyond that
users can simply write $_POST = null? We have this ability to a degree
with ZFI, when users supply $_POST to its constructor.

If we do provide this ability as a feature of a framework component,
should it be on by default? I lean toward no, since it modifies the PHP
environment by default, perhaps unexpectedly, for ZF consumers. Again,
when users supply $_POST to ZFI, by default, $_POST is set to null.

What about $_GET? There was a problem, if I recall correctly, with the
controller architecture and having set $_GET parameters to null with
ZFI... maybe it was already resolved. Matthew knows more about this than I.

I think that there must be a reasonable solution that can be reached to
meet this use case, but we obviously don't quite have it yet. I would
now leave the floor open for others to propose their ideas on how to
address this need. In the meantime, I'll do some more research and
collaboration in an effort to solve the problem.

Thanks again for your helpful feedback!

Best regards,
Darby

[Kevin McArthur-2]

> This makes a lot of sense to me. We know that we will need something
> that integrates running multiple filters and validators upon such data
> types, though with respect to the current design proposal, I would
> consider it highly related but perhaps out of scope. If this idea can
> hold, then it gives us the opportunity to propose such functionality
> separately, and I would be happy to facilitate your ideas on a design
> that would be appropriate for the framework. Maybe you would like to
> seed and/or write a proposal to illustrate how we could do this?

Darby, I've got priorities in the Zend_Pdf project unfortunatly -- I'm happy
to run ideas by and do lil patches here and there, but my roles firmly
within the Pdf stuff and I unfortunatly don't have time even for that these
days (work ft, run neutrality.ca and am writing a book under contract, so
lifes busy)

I'd suggest more thought on tying your filter/validation stuff to the
request api. The common/standarized way of handling form data would be
really useful. I'd also implement a RecursiveIterator for the collection, so
that it can handle all the right type of filtering scenarios. (Maybe
filteriterator integration?)

Right now with the framework there are too many sources of input 1)
parameters coming frm the router 2) apache environment variables 3) get
variables 4) post variables 5) cookies 6) sessions.

Consolidating these into an encapsulated environment under the Request
object might make sense.

I don't have a total solution, but the current approach seems very
stand-alone when a more integrated approach might be better?

> Yes, I see your point, and we may want to continue to offer this feature
> with the framework. I didn't really address this topic with my design
> proposal, but I think it deserves an "official" response. The questions
> to answer might be:
>
> Do we wish to provide the ability to set $_POST to null, beyond that
> users can simply write $_POST = null? We have this ability to a degree
> with ZFI, when users supply $_POST to its constructor.

I don't like this as it makes it feel non-standard and hackish to use it.

> If we do provide this ability as a feature of a framework component,
> should it be on by default? I lean toward no, since it modifies the PHP
> environment by default, perhaps unexpectedly, for ZF consumers. Again,
> when users supply $_POST to ZFI, by default, $_POST is set to null.

I'd learn towards yes, lets protect the nubes from themselves until they
learn better. I'd support an overreaching system before an underreaching one
to prevent any security problems as a result of ignorance.

> What about $_GET? There was a problem, if I recall correctly, with the
> controller architecture and having set $_GET parameters to null with
> ZFI... maybe it was already resolved. Matthew knows more about this than
> I.

Dunno about get -- thats a special case.

> I think that there must be a reasonable solution that can be reached to
> meet this use case, but we obviously don't quite have it yet. I would
> now leave the floor open for others to propose their ideas on how to
> address this need. In the meantime, I'll do some more research and
> collaboration in an effort to solve the problem.

I currently bootstrap this into my index document, and then store the result
in the registry. Any access that is needed to post data comes via the
registry and the zfi object. Keeps me happy that no one is allowing
unfiltered data into the app unless they specifically intend to with getRaw.

>
> Thanks again for your helpful feedback!

NP


K

[weierophinney]

-- Darby Felton <[hidden email]> wrote
(on Wednesday, 10 January 2007, 03:54 PM -0500):
> Kevin McArthur wrote:

Re: ZFI unsetting $_GET and/or $_POST:

> > The idea is more of social enforcement of good coding practices --
> > highly useful for team development. Direct access to post data means
> > that it will probably be used and this can cause some issues that I
> > thought ZFI solved.
>
> Yes, I see your point, and we may want to continue to offer this feature
> with the framework. I didn't really address this topic with my design
> proposal, but I think it deserves an "official" response. The questions
> to answer might be:
>
> Do we wish to provide the ability to set $_POST to null, beyond that
> users can simply write $_POST = null? We have this ability to a degree
> with ZFI, when users supply $_POST to its constructor.
>
> If we do provide this ability as a feature of a framework component,
> should it be on by default? I lean toward no, since it modifies the PHP
> environment by default, perhaps unexpectedly, for ZF consumers. Again,
> when users supply $_POST to ZFI, by default, $_POST is set to null.
>
> What about $_GET? There was a problem, if I recall correctly, with the
> controller architecture and having set $_GET parameters to null with
> ZFI... maybe it was already resolved. Matthew knows more about this than I.

Those who use ZFI to automagically filter $_GET and/or $_POST have
experienced issues with the controllers, particularly when not using
pretty urls (i.e., controller/action are $_GET or $_POST parameters). In
fact, Zend_Controller_Request_Http provides an interface to both $_GET
and $_POST, for a variety of reasons, and this breaks with a strategy
that unsets those arrays.

If we have a component that does this, then any component that needs
access to $_GET or $_POST in the framework then needs to use that
interface -- and this may be overkill or unnecessary for many of them
(think about the various server classes, for instance).

While I understand the social enforcement concept, I think it gets in
the way for those who know what they're doing, and generates confusion
for novice developers -- all the tutorials out there show accessing
$_GET and $_POST, and suddenly they're unset.

One idea I'd be interested in is the concept of re-assigning these
arrays to an object implementing ArrayAccess or extending ArrayObject
(Marco Tabini wrote about this concept in the February 2006 issue of
php|architect). Developers would still have access to the values in
normal ways, but you could perhaps set how strict that access is, and
require untainting of the data before retrieval:

$_GET = new Zend_Filter_ArrayObject($_GET, Zend_Filter_Iterator::STRICT);
echo $_GET['someKey']; // throws exception, as strict requires untainting

$_GET->untaint('someKey');
echo $_GET['someKey']; // works; value was untainted first

$_POST = new Zend_Filter_ArrayObject($_POST, Zend_Filter_Iterator::UN_STRICT);
echo $_POST['someKey']; // works; not in strict mode

Running the 'array' through a filter chain would untaint the values.

Thoughts?

--
Matthew Weier O'Phinney
PHP Developer            | [hidden email]
Zend - The PHP Company   | http://www.zend.com/

[Kevin McArthur-2]

> While I understand the social enforcement concept, I think it gets in
> the way for those who know what they're doing, and generates confusion
> for novice developers -- all the tutorials out there show accessing
> $_GET and $_POST, and suddenly they're unset.

I think you just proved _my_ point. Novice developers are confused and thus
should be protected from themselves, because otherwise it's not them who
gets blamed its us -- for being insecure, when in reality its the users.
That the tutorial says $_POST and they use it blindly should be considered a
problem _for_ doing this not against.

> One idea I'd be interested in is the concept of re-assigning these
> arrays to an object implementing ArrayAccess or extending ArrayObject
> (Marco Tabini wrote about this concept in the February 2006 issue of
> php|architect). Developers would still have access to the values in
> normal ways, but you could perhaps set how strict that access is, and
> require untainting of the data before retrieval:

I submitted a patch that made ZFI iterable in just this fashion -- it was
not accepted, because this proposal was in the works. Default for iteration
was to invoke getRaw as to not get in the way.

> $_GET = new Zend_Filter_ArrayObject($_GET, Zend_Filter_Iterator::STRICT);
> echo $_GET['someKey']; // throws exception, as strict requires untainting
>
> $_GET->untaint('someKey');
> echo $_GET['someKey']; // works; value was untainted first
>
> $_POST = new Zend_Filter_ArrayObject($_POST,
> Zend_Filter_Iterator::UN_STRICT);
> echo $_POST['someKey']; // works; not in strict mode
>
> Running the 'array' through a filter chain would untaint the values.

It'd work, but doesnt address the problems that were originally raised, like
components needing to get access to raw get and post data. Turning on strict
would just result in the framework routing failing? Or... i dont see the
benefit.

The opportunity is to create a unified, encapsulated data input system that
supports all the cool filtering/validation/detection type of stuff. I don't
see why FW components shouldnt be filtering their data anyways. To me this
would sit below the Request object in hierarchy.

Kevin

[Darby Felton]

Kevin McArthur wrote:
> I think you just proved _my_ point. Novice developers are confused and
> thus should be protected from themselves, because otherwise it's not
> them who gets blamed its us -- for being insecure, when in reality its
> the users. That the tutorial says $_POST and they use it blindly should
> be considered a problem _for_ doing this not against.

I believe that this brings us to two closely related decision points in
the Zend_Filter component design:

* Should the Zend Framework by default manipulate request variables
(e.g., $_POST) in such a way as to be incompatible with other libraries
that depend on normal superglobal behavior?

* If someone uses Zend_Filter, should they expect that code such as
"echo $_POST['someVar']" will not provide normally expected behavior?

I think that, like magic_quotes_gpc, affirming the above question would
lead to significant confusion and problems with interoperability. I
suppose it's also possible that there would be only a few "edge" or
"corner" cases with which we would have such trouble, and we could
simply document whatever incompatibility issues there are.

> The opportunity is to create a unified, encapsulated data input system
> that supports all the cool filtering/validation/detection type of stuff.
> I don't see why FW components shouldnt be filtering their data anyways.
> To me this would sit below the Request object in hierarchy.

I think what I would like to do is provide the described functionality
without imposing so much as to break too much interoperability. In order
to accomplish this, of course, we have to decide how much normal PHP
behavior we are willing to break for others, and where to do it, with
the framework.

Best regards,
Darby

[Kevin McArthur-2]

Solution:

Request->populateSuperGlobals()

let those who have compatibility problems, throw it into compatibility mode.

Kevin
----- Original Message -----
From: "Darby Felton" <[hidden email]>
To: "Kevin McArthur" <[hidden email]>
Cc: "Matthew Weier O'Phinney" <[hidden email]>; <[hidden email]>
Sent: Thursday, January 11, 2007 11:25 AM
Subject: Re: [fw-core] Re: [fw-general] Zend_Filter design proposal

> Kevin McArthur wrote:
>> I think you just proved _my_ point. Novice developers are confused and
>> thus should be protected from themselves, because otherwise it's not
>> them who gets blamed its us -- for being insecure, when in reality its
>> the users. That the tutorial says $_POST and they use it blindly should
>> be considered a problem _for_ doing this not against.
>
> I believe that this brings us to two closely related decision points in
> the Zend_Filter component design:
>
> * Should the Zend Framework by default manipulate request variables
> (e.g., $_POST) in such a way as to be incompatible with other libraries
> that depend on normal superglobal behavior?
>
> * If someone uses Zend_Filter, should they expect that code such as
> "echo $_POST['someVar']" will not provide normally expected behavior?
>
> I think that, like magic_quotes_gpc, affirming the above question would
> lead to significant confusion and problems with interoperability. I
> suppose it's also possible that there would be only a few "edge" or
> "corner" cases with which we would have such trouble, and we could
> simply document whatever incompatibility issues there are.
>
>> The opportunity is to create a unified, encapsulated data input system
>> that supports all the cool filtering/validation/detection type of stuff.
>> I don't see why FW components shouldnt be filtering their data anyways.
>> To me this would sit below the Request object in hierarchy.
>
> I think what I would like to do is provide the described functionality
> without imposing so much as to break too much interoperability. In order
> to accomplish this, of course, we have to decide how much normal PHP
> behavior we are willing to break for others, and where to do it, with
> the framework.
>
> Best regards,
> Darby

[Christopher Thompson-2]

Regarding the comments about making Zend_Filter_Input iterable, I don't
see any reason not to. That however is different from how the Filter and
Rule based chains work. This has been the difference from the beginning.
Zend_Filter_Input is essentially procedural where you process values one
by one. With the Filterchain/Validator method you provide the chain with
the Filter/Rule objects you want and then either immediately or at some
later point you process them. They both have their uses, but I for one
have a hard time building a clean Form Controller from Zend_Filter_Input.

Regarding dealing with the request superglobals, the thing you need to
understand is that the Filterchain/Validator is passed a container (e.g.
Request) and operates on that. It is not really a concern of the
Filterchain/Validator how the datasource is managed -- which is how it
should be. If you want to (or do not want to) destroy request
superglobals -- that would be a concert for a Request container.


Kevin McArthur wrote:

>> Though we recognize the need for supporting filter and validation
>> operations upon /sets/ of data, it will be some time after 0.7.0 before
>> we can fully address this complicated problem, even if only with respect
>> to form processing. Consider, for example, that the current
>> Zend_Filter_Input class operates only upon the first-level keys of its
>> input array. It does not support multidimensional arrays, where the key
>> of interest for a particular filter or validation operation may be two
>> or more levels deep into the input array. I have used multidimensional
>> arrays via $_POST often enough that I consider this to be a common use
>> case, but others' experiences may likely vary.
>
> I have successfully used Zend_Filter_Input with recursive structures
> using SPL iterator patterns. Creating a RecurisveIterator for this is
> quite simple. When I submitted the iterator patch to the old system
> (which was rejected due to this proposal being in the works) it allowed
> for treating the collection as an Iterator --- which made it easy to
> integrate with SPL recursive iterator patterns. So yes, this was sort-of
> supported.
>
>> I'm not sure that destroying raw post data was one of the original goals
>> of the Zend_Filter component, though certainly the current
>> Zend_Filter_Input does this by default if you supply $_POST to it. This
>> probably is not such a good idea for a framework component to do,
>> especially since the current component alone does not fully support
>> being handed $_POST (e.g., try using multidimensional arrays), but
>> perhaps also because interoperability issues might arise. For example,
>> if a ZF-powered app uses libraries that depend on $_POST, there could be
>> problems. I suppose one could Certainly user code can trivially set
>> $_POST = null if it is appropriate for the use case. Maybe I'm missing
>> something, though; it's happened often enough before. ;)
>
> The idea is more of social enforcement of good coding practices --
> highly useful for team development. Direct access to post data means
> that it will probably be used and this can cause some issues that I
> thought ZFI solved.
>
> Kevin
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>>
>>> While you're looking at filter/validation, another useful level might be
>>> deterministic [figguring out if a string is an email address, or
>>> contains urls etc]
>>
>> Indeed, these are good ideas; thanks for your feedback!
>>
>> Best regards,
>> Darby
>>
>>>
>>> Kevin McArthur
>>> ----- Original Message ----- From: "Darby Felton" <[hidden email]>
>>> To: "Zend Framework - General" <[hidden email]>
>>> Sent: Wednesday, January 10, 2007 10:34 AM
>>> Subject: [fw-general] Zend_Filter design proposal
>>>
>>>
>>>> Hi all,
>>>>
>>>> I have been working on a proposal for redesigning the functionality of
>>>> the current Zend_Filter component:
>>>>
>>>> http://framework.zend.com/wiki/x/Wi8
>>>>
>>>> Your review and feedback would be sincerely appreciated in this early
>>>> stage of development. Please feel free to post comments on the wiki
>>>> page
>>>> or to have your say on the fw-core mailing list, as Zend_Filter is now
>>>> considered to be within the Core infrastructure team, and I look
>>>> forward
>>>> to hearing from you!
>>>>
>>>> Subscribe to fw-core: [hidden email]
>>>>
>>>> Nabble fw-core archive: http://www.nabble.com/Zend-Core-f16191.html
>>>>
>>>> Thank you for your time and consideration!
>>>>
>>>> Best regards,
>>>> Darby
>>>
>>>
>
>
>

[Darby Felton]

Kevin McArthur wrote:
> Darby, I've got priorities in the Zend_Pdf project unfortunatly -- I'm
> happy to run ideas by and do lil patches here and there, but my roles
> firmly within the Pdf stuff and I unfortunatly don't have time even for
> that these days (work ft, run neutrality.ca and am writing a book under
> contract, so lifes busy)

Understood; I sincerely appreciate your helpful feedback! :)

> I'd suggest more thought on tying your filter/validation stuff to the
> request api. The common/standarized way of handling form data would be
> really useful. I'd also implement a RecursiveIterator for the
> collection, so that it can handle all the right type of filtering
> scenarios. (Maybe filteriterator integration?)

Aye, there's the rub. First, I have attempted to propose a design for
filters and validators such that they would support various use cases
such as filtering input (e.g., $_REQUEST), escaping output, and building
form controllers.

As for recursive iteration, I'm unsure whether it makes sense to do this
at the level of a filter or validator component. For example, if I have
an unknown number of form fields, each named "emailX", where X is a
positive integer, starting at 1 and incremented for each field, I would
want to be able to loop over each of these fields, having each value
filtered and validated the same way.

To solve this use case I think it makes sense in the form controller to
create a single set of filter and/or validator chains that can be used
with each of the "emailX" fields. That is, the same filter/validator
chain could be used for each field, a "one"-to-many relationship. In the
controller I would loop over each "emailX" field, where each field is
contained in the form's collection of fields, applying the same
filter/validator chain for each "emailX" field when needed. In this case
I think it makes sense that either a Form or Request object would
provide the container of fields to the user.

> Right now with the framework there are too many sources of input 1)
> parameters coming frm the router 2) apache environment variables 3) get
> variables 4) post variables 5) cookies 6) sessions.
>
> Consolidating these into an encapsulated environment under the Request
> object might make sense.

Yes, I see the value in offering these input data through a unified
Request object. The current Zend_Filter_Interface in the laboratory
supports this use case in the theoretical sense, perhaps akin to the
filter pattern described here:

http://www.faqs.org/docs/artu/ch11s06.html

I see no reason that a framework Request object could not make use of
filters and validators under the currently proposed design.

Perhaps we should discuss what this would solve, however. Particularly
helpful in situations like these is having use case scenarios in code
illustrate how such a Request object might be used [in an ideal world].
I guess I'm just trying to understand what such integration would solve
before trying to decide how to do design it.

> I don't have a total solution, but the current approach seems very
> stand-alone when a more integrated approach might be better?

I don't find these goals to be opposition to each other. I agree that
the currently proposed designs for filters and validators are not fully
integrated with the rest of the framework [yet], and I also agree that
we should explore the problems that such integration would solve. ZFI is
a first attempt at such integration, and I think most of us would agree
that the design needs improvement, as evidenced by your patch that
provides iteration support. But there are two different design
considerations:

(1) Design filters in such a way as to support various use case
scenarios (e.g., input filtering, escaping output) - in progress for 0.7.0

(2) Improved framework integration of filters and validation (e.g.,
input filtering), which depends on (1), planned for post-0.7.0

Best regards,
Darby

[Darby Felton]

Christopher Thompson wrote:
> Regarding the comments about making Zend_Filter_Input iterable, I don't
> see any reason not to. That however is different from how the Filter and
> Rule based chains work. This has been the difference from the beginning.
> Zend_Filter_Input is essentially procedural where you process values one
> by one. With the Filterchain/Validator method you provide the chain with
> the Filter/Rule objects you want and then either immediately or at some
> later point you process them. They both have their uses, but I for one
> have a hard time building a clean Form Controller from Zend_Filter_Input.

There may be reasons not to provide an iterator for an input filter
component, depending on how it is designed and for what uses. If
providing an iterator means that users would be able to blindly access
raw input data (or even a "filtered" version of it), this would probably
work in opposition to the goals of providing such an input filter in the
first place. I agree that we need to continue working toward a better
design for the goals Zend_Filter_Input attempts to meet using a
"mistake-proof" or "poka-yoke" approach.

> Regarding dealing with the request superglobals, the thing you need to
> understand is that the Filterchain/Validator is passed a container (e.g.
> Request) and operates on that. It is not really a concern of the
> Filterchain/Validator how the datasource is managed -- which is how it
> should be. If you want to (or do not want to) destroy request
> superglobals -- that would be a concert for a Request container.

I totally agree.

Thanks for the feedback!

Best regards,
Darby

[Kevin McArthur-2]

Darby,

All good points...

Maybe we should look at the 'form' component and how we want to handle
that -- before revising zend filter.

The goal is simple -- an extensible data processing technique for Zend
Framework.... Data comes from various sources, in a lot of different
ways..Forms are just one way right.

Basically, processing data from a CSV file needs just as much validation as
data from the Web or grabbed by a web service... Something like a gatekeeper
to sit in between Zend_Db and Zend_Request seems to make sense to me.

I'll think about the prob a bit over the weekend and i'll see what I can
come up with.

Kevin
----- Original Message -----
From: "Darby Felton" <[hidden email]>
To: "Kevin McArthur" <[hidden email]>
Cc: <[hidden email]>
Sent: Friday, January 12, 2007 11:35 AM
Subject: Re: [fw-core] Re: [fw-general] Zend_Filter design proposal

> Kevin McArthur wrote:
>> Darby, I've got priorities in the Zend_Pdf project unfortunatly -- I'm
>> happy to run ideas by and do lil patches here and there, but my roles
>> firmly within the Pdf stuff and I unfortunatly don't have time even for
>> that these days (work ft, run neutrality.ca and am writing a book under
>> contract, so lifes busy)
>
> Understood; I sincerely appreciate your helpful feedback! :)
>
>> I'd suggest more thought on tying your filter/validation stuff to the
>> request api. The common/standarized way of handling form data would be
>> really useful. I'd also implement a RecursiveIterator for the
>> collection, so that it can handle all the right type of filtering
>> scenarios. (Maybe filteriterator integration?)
>
> Aye, there's the rub. First, I have attempted to propose a design for
> filters and validators such that they would support various use cases
> such as filtering input (e.g., $_REQUEST), escaping output, and building
> form controllers.
>
> As for recursive iteration, I'm unsure whether it makes sense to do this
> at the level of a filter or validator component. For example, if I have
> an unknown number of form fields, each named "emailX", where X is a
> positive integer, starting at 1 and incremented for each field, I would
> want to be able to loop over each of these fields, having each value
> filtered and validated the same way.
>
> To solve this use case I think it makes sense in the form controller to
> create a single set of filter and/or validator chains that can be used
> with each of the "emailX" fields. That is, the same filter/validator
> chain could be used for each field, a "one"-to-many relationship. In the
> controller I would loop over each "emailX" field, where each field is
> contained in the form's collection of fields, applying the same
> filter/validator chain for each "emailX" field when needed. In this case
> I think it makes sense that either a Form or Request object would
> provide the container of fields to the user.
>
>> Right now with the framework there are too many sources of input 1)
>> parameters coming frm the router 2) apache environment variables 3) get
>> variables 4) post variables 5) cookies 6) sessions.
>>
>> Consolidating these into an encapsulated environment under the Request
>> object might make sense.
>
> Yes, I see the value in offering these input data through a unified
> Request object. The current Zend_Filter_Interface in the laboratory
> supports this use case in the theoretical sense, perhaps akin to the
> filter pattern described here:
>
> http://www.faqs.org/docs/artu/ch11s06.html
>
> I see no reason that a framework Request object could not make use of
> filters and validators under the currently proposed design.
>
> Perhaps we should discuss what this would solve, however. Particularly
> helpful in situations like these is having use case scenarios in code
> illustrate how such a Request object might be used [in an ideal world].
> I guess I'm just trying to understand what such integration would solve
> before trying to decide how to do design it.
>
>> I don't have a total solution, but the current approach seems very
>> stand-alone when a more integrated approach might be better?
>
> I don't find these goals to be opposition to each other. I agree that
> the currently proposed designs for filters and validators are not fully
> integrated with the rest of the framework [yet], and I also agree that
> we should explore the problems that such integration would solve. ZFI is
> a first attempt at such integration, and I think most of us would agree
> that the design needs improvement, as evidenced by your patch that
> provides iteration support. But there are two different design
> considerations:
>
> (1) Design filters in such a way as to support various use case
> scenarios (e.g., input filtering, escaping output) - in progress for 0.7.0
>
> (2) Improved framework integration of filters and validation (e.g.,
> input filtering), which depends on (1), planned for post-0.7.0
>
> Best regards,
> Darby

[Kevin McArthur-2]

Iteration is key -- especially for increasingly javascript/dynamic form
scenarios (which are a personal favorite of mine)

The iteration technique is designed mostly to allow for determining key
names and the number of entries. A regexp approach to this, like apply this
filter to all fields starting with email and put the results in array y,
would work just as well

Kevin
----- Original Message -----
From: "Darby Felton" <[hidden email]>
To: "Christopher Thompson" <[hidden email]>
Cc: <[hidden email]>
Sent: Friday, January 12, 2007 11:51 AM
Subject: Re: [fw-core] Re: [fw-general] Zend_Filter design proposal

> Christopher Thompson wrote:
>> Regarding the comments about making Zend_Filter_Input iterable, I don't
>> see any reason not to. That however is different from how the Filter and
>> Rule based chains work. This has been the difference from the beginning.
>> Zend_Filter_Input is essentially procedural where you process values one
>> by one. With the Filterchain/Validator method you provide the chain with
>> the Filter/Rule objects you want and then either immediately or at some
>> later point you process them. They both have their uses, but I for one
>> have a hard time building a clean Form Controller from Zend_Filter_Input.
>
> There may be reasons not to provide an iterator for an input filter
> component, depending on how it is designed and for what uses. If
> providing an iterator means that users would be able to blindly access
> raw input data (or even a "filtered" version of it), this would probably
> work in opposition to the goals of providing such an input filter in the
> first place. I agree that we need to continue working toward a better
> design for the goals Zend_Filter_Input attempts to meet using a
> "mistake-proof" or "poka-yoke" approach.
>
>> Regarding dealing with the request superglobals, the thing you need to
>> understand is that the Filterchain/Validator is passed a container (e.g.
>> Request) and operates on that. It is not really a concern of the
>> Filterchain/Validator how the datasource is managed -- which is how it
>> should be. If you want to (or do not want to) destroy request
>> superglobals -- that would be a concert for a Request container.
>
> I totally agree.
>
> Thanks for the feedback!
>
> Best regards,
> Darby
>

[Darby Felton]

Right - my point was that I'm still unclear as to where the iteration
takes place and whether you're actually iterating over filters and
validators or, for example, form fields that have filters and validators
associated with them.

Best regards,
Darby

Kevin McArthur wrote:

> Iteration is key -- especially for increasingly javascript/dynamic form
> scenarios (which are a personal favorite of mine)
>
> The iteration technique is designed mostly to allow for determining key
> names and the number of entries. A regexp approach to this, like apply
> this filter to all fields starting with email and put the results in
> array y, would work just as well
>
> Kevin
> ----- Original Message ----- From: "Darby Felton" <[hidden email]>
> To: "Christopher Thompson" <[hidden email]>
> Cc: <[hidden email]>
> Sent: Friday, January 12, 2007 11:51 AM
> Subject: Re: [fw-core] Re: [fw-general] Zend_Filter design proposal
>
>
>> Christopher Thompson wrote:
>>> Regarding the comments about making Zend_Filter_Input iterable, I don't
>>> see any reason not to. That however is different from how the Filter and
>>> Rule based chains work. This has been the difference from the beginning.
>>> Zend_Filter_Input is essentially procedural where you process values one
>>> by one. With the Filterchain/Validator method you provide the chain with
>>> the Filter/Rule objects you want and then either immediately or at some
>>> later point you process them. They both have their uses, but I for one
>>> have a hard time building a clean Form Controller from
>>> Zend_Filter_Input.
>>
>> There may be reasons not to provide an iterator for an input filter
>> component, depending on how it is designed and for what uses. If
>> providing an iterator means that users would be able to blindly access
>> raw input data (or even a "filtered" version of it), this would probably
>> work in opposition to the goals of providing such an input filter in the
>> first place. I agree that we need to continue working toward a better
>> design for the goals Zend_Filter_Input attempts to meet using a
>> "mistake-proof" or "poka-yoke" approach.
>>
>>> Regarding dealing with the request superglobals, the thing you need to
>>> understand is that the Filterchain/Validator is passed a container (e.g.
>>> Request) and operates on that. It is not really a concern of the
>>> Filterchain/Validator how the datasource is managed -- which is how it
>>> should be. If you want to (or do not want to) destroy request
>>> superglobals -- that would be a concert for a Request container.
>>
>> I totally agree.
>>
>> Thanks for the feedback!
>>
>> Best regards,
>> Darby
>>
>
>

[Kevin McArthur-2]

Doesn't really matter to me where -- just so long as you can -- I had to
patch ZFI to make it iterable and support recursive-style parsing.

----- Original Message -----
From: "Darby Felton" <[hidden email]>
To: "Kevin McArthur" <[hidden email]>
Cc: "Christopher Thompson" <[hidden email]>;
<[hidden email]>
Sent: Friday, January 12, 2007 12:03 PM
Subject: Re: [fw-core] Re: [fw-general] Zend_Filter design proposal

> Right - my point was that I'm still unclear as to where the iteration
> takes place and whether you're actually iterating over filters and
> validators or, for example, form fields that have filters and validators
> associated with them.
>
> Best regards,
> Darby
>
> Kevin McArthur wrote:
>> Iteration is key -- especially for increasingly javascript/dynamic form
>> scenarios (which are a personal favorite of mine)
>>
>> The iteration technique is designed mostly to allow for determining key
>> names and the number of entries. A regexp approach to this, like apply
>> this filter to all fields starting with email and put the results in
>> array y, would work just as well
>>
>> Kevin
>> ----- Original Message ----- From: "Darby Felton" <[hidden email]>
>> To: "Christopher Thompson" <[hidden email]>
>> Cc: <[hidden email]>
>> Sent: Friday, January 12, 2007 11:51 AM
>> Subject: Re: [fw-core] Re: [fw-general] Zend_Filter design proposal
>>
>>
>>> Christopher Thompson wrote:
>>>> Regarding the comments about making Zend_Filter_Input iterable, I don't
>>>> see any reason not to. That however is different from how the Filter
>>>> and
>>>> Rule based chains work. This has been the difference from the
>>>> beginning.
>>>> Zend_Filter_Input is essentially procedural where you process values
>>>> one
>>>> by one. With the Filterchain/Validator method you provide the chain
>>>> with
>>>> the Filter/Rule objects you want and then either immediately or at some
>>>> later point you process them. They both have their uses, but I for one
>>>> have a hard time building a clean Form Controller from
>>>> Zend_Filter_Input.
>>>
>>> There may be reasons not to provide an iterator for an input filter
>>> component, depending on how it is designed and for what uses. If
>>> providing an iterator means that users would be able to blindly access
>>> raw input data (or even a "filtered" version of it), this would probably
>>> work in opposition to the goals of providing such an input filter in the
>>> first place. I agree that we need to continue working toward a better
>>> design for the goals Zend_Filter_Input attempts to meet using a
>>> "mistake-proof" or "poka-yoke" approach.
>>>
>>>> Regarding dealing with the request superglobals, the thing you need to
>>>> understand is that the Filterchain/Validator is passed a container
>>>> (e.g.
>>>> Request) and operates on that. It is not really a concern of the
>>>> Filterchain/Validator how the datasource is managed -- which is how it
>>>> should be. If you want to (or do not want to) destroy request
>>>> superglobals -- that would be a concert for a Request container.
>>>
>>> I totally agree.
>>>
>>> Thanks for the feedback!
>>>
>>> Best regards,
>>> Darby
>>>
>>
>>

[Christopher Thompson-2]

It seems like if there are specific informational needs then adding
methods like getKeyNames() and numberOfEntries() might be a clearer
thing to do.


Kevin McArthur wrote:

> Iteration is key -- especially for increasingly javascript/dynamic form
> scenarios (which are a personal favorite of mine)
>
> The iteration technique is designed mostly to allow for determining key
> names and the number of entries. A regexp approach to this, like apply
> this filter to all fields starting with email and put the results in
> array y, would work just as well
>
> Kevin
> ----- Original Message ----- From: "Darby Felton" <[hidden email]>
> To: "Christopher Thompson" <[hidden email]>
> Cc: <[hidden email]>
> Sent: Friday, January 12, 2007 11:51 AM
> Subject: Re: [fw-core] Re: [fw-general] Zend_Filter design proposal
>
>

[Matthew Ratzloff]

> Kevin McArthur wrote:
>> I think you just proved _my_ point. Novice developers are confused and
>> thus should be protected from themselves, because otherwise it's not
>> them who gets blamed its us -- for being insecure, when in reality its
>> the users. That the tutorial says $_POST and they use it blindly should
>> be considered a problem _for_ doing this not against.
>
> I believe that this brings us to two closely related decision points in
> the Zend_Filter component design:
>
> * Should the Zend Framework by default manipulate request variables
> (e.g., $_POST) in such a way as to be incompatible with other libraries
> that depend on normal superglobal behavior?

I think it's fine to do this by default and have people go through the
Request object, so long as there's a way to TURN IT OFF and have it use
normal PHP behavior (if incorporating external projects and libraries, for
example).

-Matt