Keeping state is a good thing, but it's quite easy to make a mistake in the direction that you want to keep state in. Generally, you want to have a keep state keyword on the first rule that interacts with a packet for the connection. One common mistake that is made when mixing state tracking with filtering on flags is this:
block in all
pass in quick proto tcp from any to 20.20.20.20/32 port = 23 flags S
pass out all keep state
That certainly appears to allow a connection to be created to the telnet server on 20.20.20.20, and the replies to go back. If you try using this rule, you'll see that it does work--Momentarily. Since we're filtering for the SYN flag, the state entry never fully gets completed, and the default time to live for an incomplete state is 60 seconds.
We can solve this by rewriting the rules in one of two ways:
1)
block in all
pass in quick proto tcp from any to 20.20.20.20/32 port = 23 keep state
block out all
or:
2)
block in all
pass in quick proto tcp from any to 20.20.20.20/32 port = 23 flags S keep state
pass out all keep state
Either of these sets of rules will result in a fully established state entry for a connection to your server.