AngularJS - Modifying html element after linking

My app generates dynamic forms based on data from a web service.

For example, the web service might return: [{Name:'Age',DataType:'Number'},{Name:'PhoneNumber',DataType:'Phone'}]

In the html, the field list is bound like this:

<div ng-repeat="v in model.Variables">
    <input type="text" ng-model="v.Value" dynamic-variable="{{v.DataType}}" />
</div>

DynamicVariable is my directive that watches for changes on attrs.DynamicVariable (the value will change during linking), and performs custom actions based on the type (ex, for 'Phone' datatype, it applies a mask to the input element).

If the type is 'Number', then I have to wrap the existing element in some spans in order to create a number control that has some additional functionality.

var minusButton = '<button type="button" class="button number-down">-</button>';
var wrapper = '<span class="number input margin-right"></span>';
var plusButton = '<button type="button" class="button number-up">+</button>';
element.attr('size', '3');
element.wrap(wrapper);
element.before(minusButton);
element.after(plusButton);

However, this seems to break model binding - once the element is wrapped in more html, no binding to/from the model takes place anymore. Similarly, an element.replaceWith() approach also breaks the binding.

Any idea why this happens, or what a workaround might be? Thanks!