I like your idea very much. It is similar to the way that I prefer to estimate, with two key differences:
- I recommend a 'modified' Fibonacci sequence: 1,2, 3, 4, 5 as the point scale. 8, 13 are allowed, but represent stories too big to plan. Why do I add '4'? Because 1, 2, 3, 5 has a big hole in it...
- I use the baseline recommended by the Scaled Agile Framework (SAFe) methodology of having a one-point story representing the unit of effort required to start, execute and finish a story to the definition of done in one sprint day. We loosely define a sprint day as the six hours of an eight hour day where people actually work. Thus, we allocate eight points of work during a typical sprint for a typical team member.
That said, I am amenable to making 'half-days' a reasonably consistent substitute for full days. It is important that your decisions reflect the will and planning effectiveness of the team. Always make sure to be flexible in interpreting how agile should be implemented. Also, ensure never to freak out and get brain lock over estimates. Remember - they are ESTIMATES.